Oct 30 05:28:10.376697 ip-10-0-129-175 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 05:28:10.376705 ip-10-0-129-175 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 05:28:10.376712 ip-10-0-129-175 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 05:28:10.376931 ip-10-0-129-175 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 05:28:20.537475 ip-10-0-129-175 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 05:28:20.537493 ip-10-0-129-175 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot e4c3746545b042bd941f7c0696e75f35 -- Oct 30 05:30:20.571318 ip-10-0-129-175 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:30:20.871688 ip-10-0-129-175 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:30:20.871688 ip-10-0-129-175 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:30:20.871688 ip-10-0-129-175 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:30:20.871688 ip-10-0-129-175 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:30:20.875618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875550 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.875618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875618 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875622 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875626 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875629 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875632 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875636 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875639 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875641 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875644 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875647 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875650 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875653 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875656 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875659 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875661 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875670 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875673 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875676 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875679 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875681 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.875687 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875684 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875687 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875689 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875692 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875695 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875698 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875701 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875704 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875707 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875709 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875712 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875715 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875719 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875724 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875727 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875730 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875733 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875736 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875738 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.876168 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875741 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875743 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875746 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875748 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875751 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875754 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875756 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875759 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875763 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875765 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875768 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875771 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875773 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875776 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875778 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875782 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875785 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875787 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875790 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875792 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.876614 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875795 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875798 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875800 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875803 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875805 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875808 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875811 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875814 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875816 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875819 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875821 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875824 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875826 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875829 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875831 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875834 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875837 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875839 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875842 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.877098 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875845 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875847 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875851 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875853 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875856 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875860 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875863 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875866 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875871 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875874 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875876 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875879 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875882 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875885 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875888 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875890 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875893 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875895 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875898 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875901 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.877545 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875903 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875906 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875908 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875911 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875913 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875916 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.875918 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876693 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876700 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876703 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876707 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876710 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876713 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876716 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876719 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876722 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876725 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876728 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876730 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876733 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.878029 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876739 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876742 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876745 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876748 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876750 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876753 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876774 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876778 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876782 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876785 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876787 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876790 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876793 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876796 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876798 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876801 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876804 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876806 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876809 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.878520 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876811 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876814 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876817 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876819 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876822 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876825 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876827 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876830 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876832 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876836 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876838 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876841 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876844 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876848 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876851 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876853 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876856 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876859 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876861 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876864 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.878969 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876867 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876869 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876872 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876875 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876877 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876880 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876882 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876885 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876887 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876890 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876892 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876895 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876898 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876900 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876903 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876906 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876909 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876911 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876914 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876916 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.879463 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876919 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876921 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876927 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876930 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876932 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876937 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876939 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876942 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876945 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876948 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876950 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876953 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876955 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876958 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876961 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876963 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876966 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876969 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876971 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.879983 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876974 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876977 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876979 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876981 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876985 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876989 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876992 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876995 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.876998 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877001 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877003 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877006 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877010 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877013 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877016 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877509 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877515 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877518 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877523 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.880557 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877526 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877529 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877532 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877535 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877538 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877541 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877543 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877546 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877549 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877552 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877555 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877558 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877561 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877563 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877566 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877569 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877571 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877574 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877576 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877579 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.881198 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877581 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877584 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877587 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877589 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877592 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877595 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877598 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877600 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877603 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877606 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877609 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877613 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877616 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877619 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877621 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877624 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877627 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877629 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877632 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.881691 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877635 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877637 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877640 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877644 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877646 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877649 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877652 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877654 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877657 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877659 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877662 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877664 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877667 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877669 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877672 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877674 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877677 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877680 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877682 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877685 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.882174 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877688 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877690 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877693 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877696 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877700 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877703 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877705 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877708 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877710 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877713 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877716 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877718 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877721 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877723 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877726 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877728 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877731 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877734 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877736 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877739 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.882637 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877741 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877744 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877747 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877749 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877752 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877755 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877757 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877760 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877762 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877764 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877767 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877770 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877773 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877776 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877780 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877783 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877788 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877793 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877797 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.883127 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877800 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877803 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877806 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.877817 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878564 2581 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878575 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878582 2581 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878587 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878592 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878596 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878600 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878605 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878608 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878611 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878614 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878620 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878623 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878626 2581 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878629 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878632 2581 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878635 2581 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878638 2581 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878642 2581 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:30:20.883611 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878645 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878648 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878651 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878654 2581 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878658 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878661 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878665 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878669 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878672 2581 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878675 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878679 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878682 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878685 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878689 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878692 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878695 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878698 2581 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878701 2581 flags.go:64] FLAG: --enable-server="true" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878704 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878709 2581 flags.go:64] FLAG: --event-burst="100" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878712 2581 flags.go:64] FLAG: --event-qps="50" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878715 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878718 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878721 2581 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:30:20.884200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878728 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878731 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878734 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878737 2581 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878741 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878743 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878746 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878749 2581 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878753 2581 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878756 2581 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878759 2581 flags.go:64] FLAG: --feature-gates="" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878763 2581 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878766 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878769 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878772 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878775 2581 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878779 2581 flags.go:64] FLAG: --help="false" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878782 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-129-175.ec2.internal" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878785 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878788 2581 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878791 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878795 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878798 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878801 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:30:20.884848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878804 2581 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878807 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878810 2581 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878813 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878816 2581 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878819 2581 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878822 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878825 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878829 2581 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878832 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878835 2581 flags.go:64] FLAG: --lock-file="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878838 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878841 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878844 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878850 2581 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878853 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878856 2581 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878859 2581 flags.go:64] FLAG: --logging-format="text" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878862 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878865 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878868 2581 flags.go:64] FLAG: --manifest-url="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878871 2581 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878876 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878879 2581 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878883 2581 flags.go:64] FLAG: --max-pods="110" Oct 30 05:30:20.885488 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878887 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878890 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878893 2581 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878896 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878899 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878902 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878905 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878912 2581 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878915 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878918 2581 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878921 2581 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878924 2581 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878927 2581 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878930 2581 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878933 2581 flags.go:64] FLAG: --port="10250" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878938 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878941 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2035c58548f5275" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878944 2581 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878947 2581 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878951 2581 flags.go:64] FLAG: --register-node="true" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878954 2581 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878958 2581 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878961 2581 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878964 2581 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878966 2581 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878970 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:30:20.886129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878974 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878977 2581 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878980 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878983 2581 flags.go:64] FLAG: --runonce="false" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878986 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878990 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878993 2581 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878996 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.878999 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879003 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879006 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879009 2581 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879012 2581 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879015 2581 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879018 2581 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879021 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879024 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879027 2581 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879030 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879048 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879051 2581 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879055 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879059 2581 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879062 2581 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879065 2581 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:30:20.886746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879068 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879071 2581 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879074 2581 flags.go:64] FLAG: --v="2" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879078 2581 flags.go:64] FLAG: --version="false" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879083 2581 flags.go:64] FLAG: --vmodule="" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879087 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879091 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879244 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879248 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879251 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879255 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879258 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879260 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879263 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879266 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879270 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879272 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879275 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879278 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879281 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879284 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.887361 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879286 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879289 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879292 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879294 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879298 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879302 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879306 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879309 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879312 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879314 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879317 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879320 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879323 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879327 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879330 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879332 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879335 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879338 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.887868 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879340 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879343 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879345 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879349 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879352 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879354 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879357 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879359 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879362 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879365 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879368 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879370 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879373 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879376 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879378 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879381 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879384 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879386 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879389 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879391 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.888316 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879394 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879397 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879399 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879402 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879405 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879408 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879410 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879413 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879415 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879418 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879421 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879423 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879426 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879429 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879431 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879435 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879438 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879441 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879443 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879446 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.888833 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879448 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879451 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879454 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879457 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879459 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879462 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879465 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879467 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879470 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879473 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879475 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879478 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879481 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879483 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879486 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879488 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879491 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879494 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879496 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.889326 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879499 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879502 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879504 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879507 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879510 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879512 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879514 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879517 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879521 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879523 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879526 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879529 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879531 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879534 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.879536 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.879562 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:30:20.889782 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880365 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880382 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880388 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880393 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880403 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880407 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880411 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880416 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880420 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880425 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880429 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880437 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880442 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880446 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880451 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880455 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880465 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880470 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880474 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.890190 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880480 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880484 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880488 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880493 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880497 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880503 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880507 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880512 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880518 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880529 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880534 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880539 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880543 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880548 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880554 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880558 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880565 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880570 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.890654 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880575 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880580 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880584 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880589 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880598 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880602 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880606 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880611 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880614 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880617 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880652 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880655 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880673 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880676 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880810 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880817 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880820 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880823 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880826 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880832 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.891173 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880836 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880839 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880842 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880845 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880847 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880850 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880853 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880855 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880858 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880861 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880864 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880866 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880869 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880872 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880875 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880877 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880880 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880884 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880888 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880891 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.891648 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880894 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880896 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880899 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880901 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880904 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880906 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880909 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880912 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880914 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880917 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880919 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880925 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880927 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880930 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880932 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880935 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880938 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880940 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880943 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.892135 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880945 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880948 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880950 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880953 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880958 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880963 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880965 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880968 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880971 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.880973 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.886665 2581 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.886681 2581 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.887209 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.887216 2581 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887268 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887274 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887277 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887281 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887284 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887287 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.892623 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887290 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887293 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887296 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887299 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887301 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887304 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887307 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887310 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887313 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887315 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887318 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887321 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887324 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887326 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887329 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887332 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887335 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887337 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887340 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.893104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887343 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887345 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887348 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887352 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887355 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887358 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887361 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887364 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887366 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887369 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887372 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887375 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887377 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887380 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887383 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887385 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887388 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887390 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887393 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887396 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.893562 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887399 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887401 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887404 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887407 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887409 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887412 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887414 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887417 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887420 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887422 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887425 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887428 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887430 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887433 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887436 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887439 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887442 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887445 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887448 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887451 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.894140 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887454 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887456 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887459 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887461 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887464 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887467 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887471 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887475 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887480 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887486 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887492 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887496 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887499 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887501 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887504 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887507 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887509 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887512 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887514 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.894618 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887517 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887520 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887522 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887525 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887528 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887531 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887533 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887536 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887539 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887542 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887545 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887548 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887552 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887556 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887559 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887562 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887564 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887567 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887570 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887572 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.895109 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887575 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887577 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.887605 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887740 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887744 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887747 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887750 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887753 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887756 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887758 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887760 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887763 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887766 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887769 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887771 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887774 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.895602 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887777 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887779 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887782 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887784 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887788 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887791 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887793 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887797 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887800 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887802 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887805 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887807 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887810 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887813 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887815 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887818 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887821 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887823 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887826 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887828 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.896089 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887831 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887833 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887836 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887839 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887841 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887844 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887847 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887849 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887852 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887855 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887857 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887860 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887862 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887865 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887868 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887870 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887873 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887876 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887880 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.896610 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887884 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887888 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887890 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887894 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887896 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887899 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887901 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887904 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887906 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887909 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887912 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887914 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887917 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887919 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887922 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887924 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887927 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887930 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887932 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887935 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.897104 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887938 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887940 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887943 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887946 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887949 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887951 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887954 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887957 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887959 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887962 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887965 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887967 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887970 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887972 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887975 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887978 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887981 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887983 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887986 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887988 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.897731 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887991 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887993 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887996 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.887998 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888001 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888004 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888006 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888009 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888011 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888015 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888019 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888022 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888024 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888027 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.888062 2581 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888095 2581 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:30:20.898411 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888098 2581 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888102 2581 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888105 2581 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888107 2581 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888110 2581 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888117 2581 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888121 2581 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888124 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888126 2581 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888129 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888132 2581 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888135 2581 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888137 2581 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888140 2581 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888143 2581 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888146 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888148 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888151 2581 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888154 2581 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:30:20.898876 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888156 2581 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888159 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888162 2581 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888164 2581 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888167 2581 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888170 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888172 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888175 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888177 2581 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888180 2581 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888183 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888185 2581 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888188 2581 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888191 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888193 2581 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888196 2581 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888199 2581 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888201 2581 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888205 2581 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888208 2581 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:30:20.899482 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888211 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888213 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888216 2581 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888219 2581 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888221 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888225 2581 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888227 2581 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888230 2581 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888233 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888235 2581 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888238 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888240 2581 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888243 2581 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888245 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888248 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888250 2581 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888253 2581 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888256 2581 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888258 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:30:20.899960 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888261 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888263 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888266 2581 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888270 2581 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888274 2581 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888277 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888280 2581 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888284 2581 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888286 2581 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888289 2581 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888291 2581 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888296 2581 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888298 2581 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888301 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888304 2581 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888306 2581 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888309 2581 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888311 2581 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888314 2581 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:30:20.900431 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888317 2581 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888320 2581 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888323 2581 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888325 2581 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888328 2581 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888331 2581 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888333 2581 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888336 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888339 2581 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888341 2581 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888344 2581 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888347 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888350 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888352 2581 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888355 2581 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888357 2581 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888360 2581 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888363 2581 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888365 2581 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888368 2581 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:30:20.900909 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888370 2581 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888373 2581 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888375 2581 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888379 2581 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888382 2581 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888385 2581 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888387 2581 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:20.888390 2581 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.888443 2581 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.890256 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.891011 2581 server.go:1014] "Starting client certificate rotation" Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.891109 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.891151 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.892512 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:30:20.901467 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.893593 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:30:20.902605 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.902590 2581 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:30:20.905589 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.905566 2581 log.go:25] "Validated CRI v1 image API" Oct 30 05:30:20.907279 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.907256 2581 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:30:20.910059 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.910024 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f275cab1-49d4-415b-9415-000370bd54ab:/dev/nvme0n1p4 faf793b3-b677-47a8-beb4-fffcd7509d42:/dev/nvme0n1p3] Oct 30 05:30:20.910141 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.910058 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 05:30:20.915332 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.915220 2581 manager.go:217] Machine: {Timestamp:2026-10-30 05:30:20.914179218 +0000 UTC m=+0.254939145 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099869 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26223c2377b75c779e8eff0de3832d SystemUUID:ec26223c-2377-b75c-779e-8eff0de3832d BootID:e4c37465-45b0-42bd-941f-7c0696e75f35 Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:c9:81:61:3f Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:c9:81:61:3f Speed:0 Mtu:9001} {Name:ovs-system MacAddress:8e:90:38:a6:6a:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:30:20.915332 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.915328 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:30:20.915501 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.915452 2581 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:30:20.916340 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.916314 2581 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:30:20.916502 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.916361 2581 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-175.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:30:20.916579 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.916506 2581 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:30:20.916579 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.916519 2581 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:30:20.916579 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.916552 2581 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:30:20.916579 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.916565 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:30:20.917325 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.917311 2581 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:30:20.920891 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.920877 2581 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:30:20.921012 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921002 2581 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:30:20.921609 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921599 2581 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:30:20.921662 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921616 2581 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:30:20.921662 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921630 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:30:20.921662 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921643 2581 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:30:20.921662 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921654 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:30:20.921999 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.921986 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:30:20.923768 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.923751 2581 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:30:20.928514 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.928499 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:30:20.929171 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929157 2581 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:30:20.929223 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929190 2581 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:30:20.929358 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929347 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:30:20.929387 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929366 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:30:20.929387 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929373 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:30:20.929387 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929379 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:30:20.929387 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929384 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:30:20.929521 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929390 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:30:20.929928 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929918 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:30:20.929959 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929930 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:30:20.929959 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929937 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:30:20.929959 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929943 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:30:20.929959 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929957 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:30:20.930091 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.929966 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:30:20.930663 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.930649 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:30:20.930663 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.930663 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:30:20.934208 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.934194 2581 server.go:1263] "Started kubelet" Oct 30 05:30:20.934317 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.934291 2581 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:30:20.934443 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.934401 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:30:20.934504 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.934465 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:30:20.934902 ip-10-0-129-175 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:30:20.935662 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.935642 2581 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:30:20.936857 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.936780 2581 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:30:20.938858 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.938840 2581 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 05:30:20.941375 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.941353 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:30:20.942254 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.942239 2581 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:30:20.942828 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.942815 2581 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:30:20.942863 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.942829 2581 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:30:20.943091 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.943071 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:30:20.943236 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.943215 2581 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:30:20.943236 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.943226 2581 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:30:20.943383 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.943255 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:20.944064 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944028 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:30:20.944064 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944063 2581 factory.go:55] Registering systemd factory Oct 30 05:30:20.944212 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944073 2581 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:30:20.944283 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944272 2581 factory.go:153] Registering CRI-O factory Oct 30 05:30:20.944283 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944282 2581 factory.go:223] Registration of the crio container factory successfully Oct 30 05:30:20.944370 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944295 2581 factory.go:103] Registering Raw factory Oct 30 05:30:20.944370 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.944304 2581 manager.go:1199] Started watching for new ooms in manager Oct 30 05:30:20.945823 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.945778 2581 manager.go:319] Starting recovery of all containers Oct 30 05:30:20.956918 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.956900 2581 manager.go:324] Recovery completed Oct 30 05:30:20.962511 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.962497 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:20.965357 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965339 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:20.965414 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965371 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:20.965414 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965385 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:20.965900 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965885 2581 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:30:20.965900 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965888 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:20.965900 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965901 2581 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:30:20.966087 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.965918 2581 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:30:20.966498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.966483 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:20.966561 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.966510 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:20.966561 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.966521 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:20.968243 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.968232 2581 policy_none.go:50] "Start" Oct 30 05:30:20.968301 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.968245 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:30:20.968301 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.968255 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:30:20.970163 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.970148 2581 policy_none.go:44] "Start" Oct 30 05:30:20.973764 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.973741 2581 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 05:30:20.973986 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.973972 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-175.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 05:30:20.974518 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.973755 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-175.ec2.internal.18e3379c44e9730c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-175.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-175.ec2.internal,},FirstTimestamp:2026-10-30 05:30:20.93416526 +0000 UTC m=+0.274925191,LastTimestamp:2026-10-30 05:30:20.93416526 +0000 UTC m=+0.274925191,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-175.ec2.internal,}" Oct 30 05:30:20.986426 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.986406 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:30:20.988956 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.988931 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 05:30:20.989103 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.988997 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-175.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 05:30:20.992596 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.992573 2581 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:30:20.994380 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.994360 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7kppj" Oct 30 05:30:20.994948 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.994936 2581 manager.go:341] "Starting Device Plugin manager" Oct 30 05:30:20.995023 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.994966 2581 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:30:20.995023 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.994979 2581 server.go:87] "Starting device plugin registration server" Oct 30 05:30:20.995221 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.995208 2581 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:30:20.995275 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.995221 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:30:20.995336 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.995310 2581 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:30:20.995405 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.995396 2581 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:30:20.995405 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:20.995404 2581 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:30:20.996087 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.996070 2581 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:30:20.996162 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:20.996113 2581 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.002688 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.002666 2581 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 05:30:21.004268 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.004256 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7kppj" Oct 30 05:30:21.061087 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.061066 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:30:21.063142 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.062248 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:30:21.063142 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.062643 2581 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:30:21.063142 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.062668 2581 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:30:21.063142 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.062703 2581 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 05:30:21.076860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.076843 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:21.095883 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.095867 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:21.096672 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.096657 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:21.096739 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.096685 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:21.096739 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.096699 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:21.096739 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.096726 2581 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.113772 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.113755 2581 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.113845 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.113773 2581 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-175.ec2.internal\": node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.155669 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.155647 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.163398 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.163376 2581 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal"] Oct 30 05:30:21.163462 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.163428 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:21.164215 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.164197 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:21.164279 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.164223 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:21.164279 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.164236 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:21.166430 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.166418 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:21.167518 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.166864 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.167518 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.166910 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:21.167658 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.167602 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:21.167658 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.167626 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:21.167658 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.167642 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:21.168372 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.168353 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:21.168465 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.168436 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:21.168732 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.168720 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:21.170884 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.170868 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.170954 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.170890 2581 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:30:21.171578 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.171559 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:30:21.171658 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.171585 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:30:21.171658 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.171595 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:30:21.201518 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.201503 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-175.ec2.internal\" not found" node="ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.204210 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.204196 2581 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-175.ec2.internal\" not found" node="ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.244699 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.244675 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.244779 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.244707 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.244779 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.244723 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44ac8e1d4fed32053c2639c09d7471d6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" (UID: \"44ac8e1d4fed32053c2639c09d7471d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.255744 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.255729 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.345018 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.344997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.345111 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.345021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.345111 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.345050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44ac8e1d4fed32053c2639c09d7471d6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" (UID: \"44ac8e1d4fed32053c2639c09d7471d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.345111 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.345104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.345201 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.345111 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.345201 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.345146 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44ac8e1d4fed32053c2639c09d7471d6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" (UID: \"44ac8e1d4fed32053c2639c09d7471d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.355911 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.355890 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.456464 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.456423 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.505103 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.505079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.507222 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.507209 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:30:21.556851 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.556726 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.657221 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.657200 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.757697 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.757654 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.858138 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.858115 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:21.891298 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.891274 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 05:30:21.891664 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:21.891415 2581 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 05:30:21.959153 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:21.959129 2581 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:30:22.009398 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.009343 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:25:22 +0000 UTC" deadline="2028-05-02 17:46:53.815962689 +0000 UTC" Oct 30 05:30:22.009398 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.009369 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13212h16m31.806596134s" Oct 30 05:30:22.032009 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.031988 2581 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:22.044011 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.043989 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:30:22.045355 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.045337 2581 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:22.061394 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:22.061367 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ac8e1d4fed32053c2639c09d7471d6.slice/crio-c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460 WatchSource:0}: Error finding container c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460: Status 404 returned error can't find the container with id c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460 Oct 30 05:30:22.061915 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:22.061893 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63e598fb2c0ecb0a10670947e4c1ef09.slice/crio-7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853 WatchSource:0}: Error finding container 7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853: Status 404 returned error can't find the container with id 7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853 Oct 30 05:30:22.066296 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.066241 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" event={"ID":"63e598fb2c0ecb0a10670947e4c1ef09","Type":"ContainerStarted","Data":"7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853"} Oct 30 05:30:22.066383 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.066279 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:30:22.067602 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.067585 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" event={"ID":"44ac8e1d4fed32053c2639c09d7471d6","Type":"ContainerStarted","Data":"c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460"} Oct 30 05:30:22.080128 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.080112 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:22.080938 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.080921 2581 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:30:22.111946 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:22.111927 2581 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:30:23.749866 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:23.749821 2581 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:24.148996 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.148964 2581 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:30:24.317204 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.317172 2581 apiserver.go:52] "Watching apiserver" Oct 30 05:30:24.337209 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.337182 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 05:30:24.357845 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.357818 2581 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:30:24.358471 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.358141 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-rkb2j","openshift-network-diagnostics/network-check-target-lqd5k","openshift-cluster-node-tuning-operator/tuned-w4sqk","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal","openshift-multus/multus-tbp8n","openshift-network-operator/iptables-alerter-dx7wt","openshift-ovn-kubernetes/ovnkube-node-x887j","kube-system/global-pull-secret-syncer-rzb85","kube-system/konnectivity-agent-lh482","openshift-dns/node-resolver-8wf8z","openshift-multus/multus-additional-cni-plugins-d5q8v","kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn","openshift-image-registry/node-ca-gbpks"] Oct 30 05:30:24.360798 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.360777 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.363009 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.362987 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.365140 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.365121 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.368796 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.368770 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.371002 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.370846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.373004 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.372986 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.375574 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.375308 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.380464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.380440 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.382786 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.382766 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nnkr\"" Oct 30 05:30:24.382896 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.382882 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.382956 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.382925 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.382956 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.382925 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-76x48\"" Oct 30 05:30:24.383055 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.383028 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.383107 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.383095 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.383301 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.383243 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:30:24.385292 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.385275 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:24.385381 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.385342 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:24.388146 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.388108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:24.388255 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.388187 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:24.390606 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.390583 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.392750 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.392728 2581 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 05:30:24.392868 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.392788 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nf99z" Oct 30 05:30:24.392987 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.392965 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.393048 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.393034 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:24.404876 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.404858 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nf99z" Oct 30 05:30:24.439234 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.439214 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:30:24.447758 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k454\" (UniqueName: \"kubernetes.io/projected/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-kube-api-access-4k454\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:24.447869 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysconfig\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.447869 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-k8s-cni-cncf-io\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.447869 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447825 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-bin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.447869 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447850 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-var-lib-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.447869 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447868 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dns2\" (UniqueName: \"kubernetes.io/projected/4d0513be-2542-402d-bc28-d4fdea2084f1-kube-api-access-9dns2\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447905 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-netns\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447943 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-conf-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-daemon-config\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.447986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmjc5\" (UniqueName: \"kubernetes.io/projected/09b60a4b-f264-48b1-a862-f5a54b7316da-kube-api-access-bmjc5\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93637adc-fd32-44b0-89ba-9a26152b5c79-ovn-node-metrics-cert\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448014 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-system-cni-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448038 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448054 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42226e87-0da6-4fa1-8d6c-842368393e3e-serviceca\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448074 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-tmp\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448089 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eddd45fc-666b-4ba0-a049-10623018ea60-agent-certs\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.448116 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448103 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-slash\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw8f6\" (UniqueName: \"kubernetes.io/projected/93637adc-fd32-44b0-89ba-9a26152b5c79-kube-api-access-pw8f6\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448143 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-cnibin\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448163 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-tmp-dir\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448177 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-device-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-etc-selinux\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448208 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-host\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448227 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn6rx\" (UniqueName: \"kubernetes.io/projected/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-kube-api-access-xn6rx\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448255 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-node-log\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448279 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-socket-dir-parent\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glnts\" (UniqueName: \"kubernetes.io/projected/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-kube-api-access-glnts\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448341 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-conf\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448370 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448392 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-config\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448551 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:24.448742 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448578 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-run\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-sys\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-multus\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448666 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-etc-kubernetes\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-script-lib\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448717 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-tuned\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448735 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-multus-certs\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448761 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-log-socket\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448789 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-kubelet-config\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-dbus\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448833 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-systemd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448874 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448889 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-netd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448904 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-env-overrides\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-hosts-file\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448939 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-kubernetes\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448954 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-systemd\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.449464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448969 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-host-slash\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.448985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-kubelet\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-socket-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449047 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-lib-modules\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449067 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nns62\" (UniqueName: \"kubernetes.io/projected/e30d6d09-f65b-4eb7-b964-7698457935cc-kube-api-access-nns62\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449082 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-system-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449097 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-kubelet\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449119 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eddd45fc-666b-4ba0-a049-10623018ea60-konnectivity-ca\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449136 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-etc-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449152 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-modprobe-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449167 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-iptables-alerter-script\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449181 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-ovn\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449195 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-bin\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449210 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449260 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42226e87-0da6-4fa1-8d6c-842368393e3e-host\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.450255 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449275 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-var-lib-kubelet\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449351 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-cni-binary-copy\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-systemd-units\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449383 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449397 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-hostroot\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449413 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449428 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-registration-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449443 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-sys-fs\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449458 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r482q\" (UniqueName: \"kubernetes.io/projected/42226e87-0da6-4fa1-8d6c-842368393e3e-kube-api-access-r482q\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449480 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-cnibin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449500 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7s6\" (UniqueName: \"kubernetes.io/projected/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kube-api-access-lk7s6\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449523 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449570 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-netns\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.451050 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.449598 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.481078 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481050 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.481175 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481097 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.481248 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481191 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:30:24.481248 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481189 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:30:24.481248 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481232 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.481248 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481242 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481280 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481305 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481307 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481322 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481339 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481345 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481390 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481397 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481412 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481414 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:30:24.481438 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481440 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w87jr\"" Oct 30 05:30:24.481788 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481455 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-88sr6\"" Oct 30 05:30:24.481788 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481469 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.481788 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.481494 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:30:24.493735 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.493714 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cj6p\"" Oct 30 05:30:24.493843 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.493741 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twkbr\"" Oct 30 05:30:24.493843 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.493762 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:30:24.493843 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.493787 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:30:24.496642 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.494614 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wnglw\"" Oct 30 05:30:24.496642 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.494623 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:30:24.496642 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.494777 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c6sjw\"" Oct 30 05:30:24.496642 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.494975 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt4k5\"" Oct 30 05:30:24.496642 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.495266 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:30:24.549831 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549803 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-kubelet-config\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.549934 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-dbus\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.549934 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549869 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-systemd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.549934 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549910 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-systemd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.549934 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549913 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549948 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-netd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-env-overrides\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549979 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.549997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-hosts-file\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550022 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-kubernetes\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550047 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-systemd\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550058 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-dbus\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550089 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-kubelet-config\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-systemd\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550025 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-netd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550141 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-kubernetes\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550157 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550159 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-host-slash\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550165 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-hosts-file\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-kubelet\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550191 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-socket-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550204 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-lib-modules\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550203 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-host-slash\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nns62\" (UniqueName: \"kubernetes.io/projected/e30d6d09-f65b-4eb7-b964-7698457935cc-kube-api-access-nns62\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550256 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-lib-modules\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550256 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-system-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-kubelet\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550293 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eddd45fc-666b-4ba0-a049-10623018ea60-konnectivity-ca\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550294 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-system-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550307 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-etc-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550322 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-modprobe-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-iptables-alerter-script\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550350 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-ovn\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550364 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-bin\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.550718 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550381 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550398 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-env-overrides\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550435 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42226e87-0da6-4fa1-8d6c-842368393e3e-host\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550470 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-var-lib-kubelet\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-cni-binary-copy\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550512 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-kubelet\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550521 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-systemd-units\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.550548 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550553 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-ovn\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-systemd-units\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550608 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-etc-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-socket-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-var-lib-kubelet\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550513 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-bin\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550628 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-modprobe-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.551498 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.550624 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:25.050590382 +0000 UTC m=+2.996857265 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550497 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-kubelet\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550668 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550694 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-hostroot\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550719 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-registration-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-sys-fs\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550775 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r482q\" (UniqueName: \"kubernetes.io/projected/42226e87-0da6-4fa1-8d6c-842368393e3e-kube-api-access-r482q\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550840 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-sys-fs\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550844 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-registration-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550874 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-hostroot\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550882 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-cnibin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550924 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42226e87-0da6-4fa1-8d6c-842368393e3e-host\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-cnibin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550958 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7s6\" (UniqueName: \"kubernetes.io/projected/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kube-api-access-lk7s6\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.550997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:24.552052 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-netns\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551053 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551079 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k454\" (UniqueName: \"kubernetes.io/projected/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-kube-api-access-4k454\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551095 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-iptables-alerter-script\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551103 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysconfig\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551133 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-k8s-cni-cncf-io\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551158 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551178 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eddd45fc-666b-4ba0-a049-10623018ea60-konnectivity-ca\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551201 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-netns\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-k8s-cni-cncf-io\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551248 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysconfig\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-cni-binary-copy\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551266 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551277 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-bin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551302 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-var-lib-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dns2\" (UniqueName: \"kubernetes.io/projected/4d0513be-2542-402d-bc28-d4fdea2084f1-kube-api-access-9dns2\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.552744 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551350 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-var-lib-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551318 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-bin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551362 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-netns\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-conf-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551433 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-daemon-config\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551456 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjc5\" (UniqueName: \"kubernetes.io/projected/09b60a4b-f264-48b1-a862-f5a54b7316da-kube-api-access-bmjc5\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551473 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93637adc-fd32-44b0-89ba-9a26152b5c79-ovn-node-metrics-cert\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551514 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-system-cni-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551556 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551585 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42226e87-0da6-4fa1-8d6c-842368393e3e-serviceca\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551590 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-system-cni-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-tmp\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eddd45fc-666b-4ba0-a049-10623018ea60-agent-certs\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551656 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-slash\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551679 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8f6\" (UniqueName: \"kubernetes.io/projected/93637adc-fd32-44b0-89ba-9a26152b5c79-kube-api-access-pw8f6\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.553473 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551702 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-cnibin\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-tmp-dir\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551749 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-device-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551773 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-etc-selinux\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551762 2581 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551797 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-host\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551803 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551821 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6rx\" (UniqueName: \"kubernetes.io/projected/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-kube-api-access-xn6rx\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-node-log\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551871 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-socket-dir-parent\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551896 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnts\" (UniqueName: \"kubernetes.io/projected/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-kube-api-access-glnts\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551918 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-conf\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551973 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-config\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.551997 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552021 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552043 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-run\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-sys\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.554230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-multus\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552101 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-etc-kubernetes\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-script-lib\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552163 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-tuned\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552189 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-multus-certs\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552212 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-log-socket\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552279 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-log-socket\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552334 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-slash\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552544 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-cnibin\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552668 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-conf-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552784 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-multus\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552849 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.552920 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.552953 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42226e87-0da6-4fa1-8d6c-842368393e3e-serviceca\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.552960 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:25.052945838 +0000 UTC m=+2.999212700 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553001 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-run\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.555029 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553044 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-sys\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553104 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-tmp-dir\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553148 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-device-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553169 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-node-log\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553206 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-socket-dir-parent\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553230 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-etc-kubernetes\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-etc-selinux\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553301 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-host\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-conf\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553561 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553613 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-daemon-config\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553681 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-multus-certs\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553723 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-netns\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553838 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-config\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.555860 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.553981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-script-lib\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.556492 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.555963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-tmp\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.556492 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.556076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93637adc-fd32-44b0-89ba-9a26152b5c79-ovn-node-metrics-cert\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.556606 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.556559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eddd45fc-666b-4ba0-a049-10623018ea60-agent-certs\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.556918 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.556895 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-tuned\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.572513 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.572493 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:24.572513 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.572514 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:24.572700 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.572538 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:24.572700 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:24.572608 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:30:25.072588638 +0000 UTC m=+3.018855518 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:24.575782 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.575737 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8f6\" (UniqueName: \"kubernetes.io/projected/93637adc-fd32-44b0-89ba-9a26152b5c79-kube-api-access-pw8f6\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.576591 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.576548 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dns2\" (UniqueName: \"kubernetes.io/projected/4d0513be-2542-402d-bc28-d4fdea2084f1-kube-api-access-9dns2\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.576690 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.576573 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k454\" (UniqueName: \"kubernetes.io/projected/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-kube-api-access-4k454\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:24.578101 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.578081 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nns62\" (UniqueName: \"kubernetes.io/projected/e30d6d09-f65b-4eb7-b964-7698457935cc-kube-api-access-nns62\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.578364 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.578347 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjc5\" (UniqueName: \"kubernetes.io/projected/09b60a4b-f264-48b1-a862-f5a54b7316da-kube-api-access-bmjc5\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.581113 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.581083 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r482q\" (UniqueName: \"kubernetes.io/projected/42226e87-0da6-4fa1-8d6c-842368393e3e-kube-api-access-r482q\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.581570 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.581551 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7s6\" (UniqueName: \"kubernetes.io/projected/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kube-api-access-lk7s6\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.582191 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.582143 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6rx\" (UniqueName: \"kubernetes.io/projected/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-kube-api-access-xn6rx\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.586342 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.586323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnts\" (UniqueName: \"kubernetes.io/projected/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-kube-api-access-glnts\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.674066 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.674000 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:30:24.683246 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.683217 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67e26d01_cd9d_4e17_bd57_f7eb78cdc92e.slice/crio-17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907 WatchSource:0}: Error finding container 17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907: Status 404 returned error can't find the container with id 17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907 Oct 30 05:30:24.687228 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.687204 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:30:24.693372 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.693348 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:30:24.698523 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.698501 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6b480bd_5b3f_494c_a6e8_b59b15ba4818.slice/crio-73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208 WatchSource:0}: Error finding container 73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208: Status 404 returned error can't find the container with id 73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208 Oct 30 05:30:24.699354 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.699331 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:30:24.705240 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.705219 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-tbp8n" Oct 30 05:30:24.711037 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.711013 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42226e87_0da6_4fa1_8d6c_842368393e3e.slice/crio-e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573 WatchSource:0}: Error finding container e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573: Status 404 returned error can't find the container with id e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573 Oct 30 05:30:24.713573 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.713521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:30:24.716318 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.716288 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09b60a4b_f264_48b1_a862_f5a54b7316da.slice/crio-d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71 WatchSource:0}: Error finding container d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71: Status 404 returned error can't find the container with id d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71 Oct 30 05:30:24.719151 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.719139 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:24.723928 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.723898 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5234bc_f0a0_4f7f_a5b4_3443c75e274a.slice/crio-bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554 WatchSource:0}: Error finding container bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554: Status 404 returned error can't find the container with id bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554 Oct 30 05:30:24.730048 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.730016 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeddd45fc_666b_4ba0_a049_10623018ea60.slice/crio-d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872 WatchSource:0}: Error finding container d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872: Status 404 returned error can't find the container with id d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872 Oct 30 05:30:24.749765 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.749742 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:24.756087 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:24.756067 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:30:24.764284 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.764252 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93637adc_fd32_44b0_89ba_9a26152b5c79.slice/crio-408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776 WatchSource:0}: Error finding container 408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776: Status 404 returned error can't find the container with id 408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776 Oct 30 05:30:24.769788 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:30:24.769757 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0513be_2542_402d_bc28_d4fdea2084f1.slice/crio-32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d WatchSource:0}: Error finding container 32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d: Status 404 returned error can't find the container with id 32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d Oct 30 05:30:25.056091 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.055571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:25.056091 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.055818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:25.056091 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.055741 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:25.056091 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.056028 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:26.056009379 +0000 UTC m=+4.002276261 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:25.056091 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.055961 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:25.056091 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.056068 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:26.056058777 +0000 UTC m=+4.002325655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:25.157769 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.157162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:25.157769 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.157350 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:25.157769 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.157368 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:25.157769 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.157381 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:25.157769 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:25.157435 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:30:26.157417658 +0000 UTC m=+4.103684534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:25.407730 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.407647 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:24 +0000 UTC" deadline="2028-07-10 07:12:35.16679446 +0000 UTC" Oct 30 05:30:25.407730 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.407689 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14857h42m9.759107975s" Oct 30 05:30:25.469865 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.469749 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" event={"ID":"e30d6d09-f65b-4eb7-b964-7698457935cc","Type":"ContainerStarted","Data":"224c0c505ef1d5857974f9b243d96fe4c2aa33bff4e3ac888fd6bd442c6221b9"} Oct 30 05:30:25.474095 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.474022 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wf8z" event={"ID":"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e","Type":"ContainerStarted","Data":"17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907"} Oct 30 05:30:25.481035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.480582 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" event={"ID":"44ac8e1d4fed32053c2639c09d7471d6","Type":"ContainerStarted","Data":"19ab01aa9531f398644ed6b550cac740950a98b4b956542dbab75ee063419555"} Oct 30 05:30:25.483007 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.482857 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776"} Oct 30 05:30:25.488252 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.488173 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dx7wt" event={"ID":"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a","Type":"ContainerStarted","Data":"bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554"} Oct 30 05:30:25.490988 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.490911 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbp8n" event={"ID":"09b60a4b-f264-48b1-a862-f5a54b7316da","Type":"ContainerStarted","Data":"d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71"} Oct 30 05:30:25.495786 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.495759 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" event={"ID":"c6b480bd-5b3f-494c-a6e8-b59b15ba4818","Type":"ContainerStarted","Data":"73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208"} Oct 30 05:30:25.514537 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.511878 2581 generic.go:356] "Generic (PLEG): container finished" podID="63e598fb2c0ecb0a10670947e4c1ef09" containerID="3d30e6744beac1ef0d3dd1a4ec171e397168d27324800d036cc0cae619f6a57f" exitCode=0 Oct 30 05:30:25.514537 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.512151 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" event={"ID":"63e598fb2c0ecb0a10670947e4c1ef09","Type":"ContainerDied","Data":"3d30e6744beac1ef0d3dd1a4ec171e397168d27324800d036cc0cae619f6a57f"} Oct 30 05:30:25.518012 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.517986 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerStarted","Data":"32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d"} Oct 30 05:30:25.525367 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.525319 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" podStartSLOduration=2.5253027340000003 podStartE2EDuration="2.525302734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:25.524822521 +0000 UTC m=+3.471089409" watchObservedRunningTime="2026-10-30 05:30:25.525302734 +0000 UTC m=+3.471569680" Oct 30 05:30:25.528971 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.528926 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lh482" event={"ID":"eddd45fc-666b-4ba0-a049-10623018ea60","Type":"ContainerStarted","Data":"d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872"} Oct 30 05:30:25.536482 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:25.536418 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gbpks" event={"ID":"42226e87-0da6-4fa1-8d6c-842368393e3e","Type":"ContainerStarted","Data":"e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573"} Oct 30 05:30:26.065817 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.065751 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:26.065994 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.065831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:26.065994 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.065971 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:26.066103 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.066028 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:28.066008382 +0000 UTC m=+6.012275262 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:26.066421 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.066404 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:26.066497 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.066469 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:28.066442495 +0000 UTC m=+6.012709375 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:26.167136 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.167091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:26.167300 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.167232 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:26.167300 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.167253 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:26.167300 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.167264 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:26.167451 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.167313 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:30:28.167297268 +0000 UTC m=+6.113564131 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:26.408033 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.407941 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:24 +0000 UTC" deadline="2028-04-11 10:43:18.105042357 +0000 UTC" Oct 30 05:30:26.408033 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.407986 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12701h12m51.697060335s" Oct 30 05:30:26.459797 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.459769 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:26.459965 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.459890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:26.460325 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.460299 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:26.460436 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.460417 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:26.460499 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.460482 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:26.460579 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:26.460568 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:26.548504 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:26.548411 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" event={"ID":"63e598fb2c0ecb0a10670947e4c1ef09","Type":"ContainerStarted","Data":"3acd4c554341a466a771b9535aa1dfadc0c90573342a658a789f6415964ddf8d"} Oct 30 05:30:27.560209 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:27.558876 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="7186d1d3b2a9561ffb28c4cf12104bda5b2524dc9f8e7a0ab0a6692a86c61771" exitCode=0 Oct 30 05:30:27.560209 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:27.559926 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerDied","Data":"7186d1d3b2a9561ffb28c4cf12104bda5b2524dc9f8e7a0ab0a6692a86c61771"} Oct 30 05:30:27.631137 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:27.630797 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" podStartSLOduration=4.630776052 podStartE2EDuration="4.630776052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:26.588076861 +0000 UTC m=+4.534343747" watchObservedRunningTime="2026-10-30 05:30:27.630776052 +0000 UTC m=+5.577042939" Oct 30 05:30:28.081629 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:28.081589 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:28.081797 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:28.081659 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:28.081864 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.081792 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:28.081864 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.081858 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.081838357 +0000 UTC m=+10.028105236 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:28.082278 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.082257 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:28.082400 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.082302 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.082291395 +0000 UTC m=+10.028558272 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:28.182130 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:28.182090 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:28.182317 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.182302 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:28.182380 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.182325 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:28.182380 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.182339 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:28.182479 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.182393 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:30:32.182377027 +0000 UTC m=+10.128643904 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:28.460289 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:28.460050 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:28.460289 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.460173 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:28.460625 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:28.460575 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:28.460715 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.460670 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:28.460764 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:28.460730 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:28.460920 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:28.460806 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:29.565973 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:29.565937 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-lh482" event={"ID":"eddd45fc-666b-4ba0-a049-10623018ea60","Type":"ContainerStarted","Data":"d27e3955ed6c054a0301aa1ee24df55e9e5259bef9b21ba2051782459a72b561"} Oct 30 05:30:29.606501 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:29.606435 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-lh482" podStartSLOduration=3.7291344239999997 podStartE2EDuration="7.606416449s" totalImagesPullingTime="3.877282025s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:29.605717594 +0000 UTC m=+7.551984479" watchObservedRunningTime="2026-10-30 05:30:29.606416449 +0000 UTC m=+7.552683336" Oct 30 05:30:30.457762 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:30.457691 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:30.457912 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:30.457816 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:30.457912 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:30.457879 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:30.458035 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:30.457954 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:30.458035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:30.457995 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:30.458134 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:30.458061 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:30.571013 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:30.570984 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gbpks" event={"ID":"42226e87-0da6-4fa1-8d6c-842368393e3e","Type":"ContainerStarted","Data":"c596a3d618f512bf3d8c9ea047ab3344eb58ef664287c9eea1db1444b32fc0de"} Oct 30 05:30:31.578618 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:31.578522 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" event={"ID":"c6b480bd-5b3f-494c-a6e8-b59b15ba4818","Type":"ContainerStarted","Data":"31d05038d364027b2795ffaeac5bf8504c8dd62d4493e73fdd26667769add541"} Oct 30 05:30:32.024510 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.024483 2581 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:30:32.065858 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.065824 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:32.066619 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.066600 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:32.110345 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.110287 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gbpks" podStartSLOduration=5.192046186 podStartE2EDuration="10.110270851s" totalImagesPullingTime="4.918224665s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:30.609910167 +0000 UTC m=+8.556177051" watchObservedRunningTime="2026-10-30 05:30:32.110270851 +0000 UTC m=+10.056537747" Oct 30 05:30:32.112367 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.112337 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:32.112477 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.112441 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:32.112819 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.112797 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:32.112928 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.112883 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:40.112863839 +0000 UTC m=+18.059130712 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:32.113186 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.113169 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:32.113274 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.113210 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:40.113198053 +0000 UTC m=+18.059464931 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:32.213460 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.213168 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:32.213460 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.213335 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:32.213460 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.213354 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:32.213460 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.213367 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:32.213460 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.213419 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:30:40.213401412 +0000 UTC m=+18.159668277 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:32.408289 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.407572 2581 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:30:32.024504462Z","UUID":"af1a94c5-abd9-46ed-a08a-8854318ce457","Handler":null,"Name":"","Endpoint":""} Oct 30 05:30:32.409933 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.409441 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:30:32.409933 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.409470 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:30:32.458407 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.458353 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:32.458555 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.458483 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:32.458649 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.458630 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:32.458773 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.458733 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:32.458841 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.458796 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:32.458896 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:32.458877 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:32.582272 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.582219 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-tbp8n" event={"ID":"09b60a4b-f264-48b1-a862-f5a54b7316da","Type":"ContainerStarted","Data":"bb4b4118d39f033260f8ab5403c134292ae0b0005d31e5b5dc37e600fc01543b"} Oct 30 05:30:32.586913 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.586837 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" event={"ID":"c6b480bd-5b3f-494c-a6e8-b59b15ba4818","Type":"ContainerStarted","Data":"2abcecb71f02a65ca6eb96237303009cef726f2c69db0cca07156f335952ec4e"} Oct 30 05:30:32.589334 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.589308 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="7f0a08e102dd06b8032200b39a622493b66f1004f68909b325118c5a1f203ab9" exitCode=0 Oct 30 05:30:32.589572 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.589499 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerDied","Data":"7f0a08e102dd06b8032200b39a622493b66f1004f68909b325118c5a1f203ab9"} Oct 30 05:30:32.627554 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:32.627153 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-tbp8n" podStartSLOduration=3.209338549 podStartE2EDuration="10.627140705s" totalImagesPullingTime="7.417802156s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:32.626922126 +0000 UTC m=+10.573189012" watchObservedRunningTime="2026-10-30 05:30:32.627140705 +0000 UTC m=+10.573407590" Oct 30 05:30:33.594805 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:33.594521 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="1aea8620247dbba37e499e958dd802b5b26a4578f5b884461bc111ec41dc045a" exitCode=0 Oct 30 05:30:33.594805 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:33.594594 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerDied","Data":"1aea8620247dbba37e499e958dd802b5b26a4578f5b884461bc111ec41dc045a"} Oct 30 05:30:33.597318 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:33.597289 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8wf8z" event={"ID":"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e","Type":"ContainerStarted","Data":"ec94350eaa582929c14f45dc21dff0188b695c0112dc661c7bbe0617902dbfe0"} Oct 30 05:30:33.600624 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:33.600582 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" event={"ID":"c6b480bd-5b3f-494c-a6e8-b59b15ba4818","Type":"ContainerStarted","Data":"feeeb3bfc706117fdd19fe721452378967641a883738e9842f467f55661a6703"} Oct 30 05:30:33.681895 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:33.681846 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-8wf8z" podStartSLOduration=3.794432349 podStartE2EDuration="11.681828554s" totalImagesPullingTime="7.887396205s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:33.681249042 +0000 UTC m=+11.627515927" watchObservedRunningTime="2026-10-30 05:30:33.681828554 +0000 UTC m=+11.628095442" Oct 30 05:30:33.722912 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:33.722861 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" podStartSLOduration=3.9138613920000003 podStartE2EDuration="11.722849544s" totalImagesPullingTime="7.808988152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:33.722316548 +0000 UTC m=+11.668583444" watchObservedRunningTime="2026-10-30 05:30:33.722849544 +0000 UTC m=+11.669116429" Oct 30 05:30:34.457782 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.457754 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:34.457930 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:34.457871 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:34.458230 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.458202 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:34.458344 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:34.458303 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:34.458482 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.458458 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:34.458624 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:34.458599 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:34.604263 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.604187 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dx7wt" event={"ID":"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a","Type":"ContainerStarted","Data":"f659fff10f6e264a0bd1e758786c668002f91d03c9a2ae85f42b7536c8078fc7"} Oct 30 05:30:34.607270 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.606777 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="2a7f078badb7d21f0fba642f69e444f45543f9b622d681c6de48d893d522b26d" exitCode=0 Oct 30 05:30:34.607270 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.606869 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerDied","Data":"2a7f078badb7d21f0fba642f69e444f45543f9b622d681c6de48d893d522b26d"} Oct 30 05:30:34.608792 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.608700 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" event={"ID":"e30d6d09-f65b-4eb7-b964-7698457935cc","Type":"ContainerStarted","Data":"78657dfd40502604292c5ee235f7c1e2a49cff63aa88b79855c0280635149e7b"} Oct 30 05:30:34.681167 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.681112 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dx7wt" podStartSLOduration=5.106447077 podStartE2EDuration="12.681096331s" totalImagesPullingTime="7.574649254s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:34.680648267 +0000 UTC m=+12.626915153" watchObservedRunningTime="2026-10-30 05:30:34.681096331 +0000 UTC m=+12.627363217" Oct 30 05:30:34.768084 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:34.768009 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" podStartSLOduration=3.194079376 podStartE2EDuration="12.767993954s" totalImagesPullingTime="9.573914578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:34.767638406 +0000 UTC m=+12.713905292" watchObservedRunningTime="2026-10-30 05:30:34.767993954 +0000 UTC m=+12.714260840" Oct 30 05:30:36.457850 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:36.457776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:36.458514 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:36.457776 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:36.458514 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:36.457907 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:36.458514 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:36.457922 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:36.458514 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:36.458012 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:36.458514 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:36.458102 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:36.616506 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:36.616470 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="61db0e09b3acccbd1bf2a19ba1f1381519c92f2357f0b1058ec547d57a88b59e" exitCode=0 Oct 30 05:30:36.616669 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:36.616523 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerDied","Data":"61db0e09b3acccbd1bf2a19ba1f1381519c92f2357f0b1058ec547d57a88b59e"} Oct 30 05:30:37.623494 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:37.623461 2581 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="b8c60550f375e7ee93c448fd3b1737133fe9908d3c39c38c0a64fa9517f46992" exitCode=0 Oct 30 05:30:37.623962 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:37.623514 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerDied","Data":"b8c60550f375e7ee93c448fd3b1737133fe9908d3c39c38c0a64fa9517f46992"} Oct 30 05:30:38.457656 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:38.457621 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:38.457656 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:38.457654 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:38.457881 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:38.457695 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:38.457881 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:38.457786 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:38.457969 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:38.457913 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:38.458037 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:38.458008 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:38.628416 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:38.628385 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" event={"ID":"4d0513be-2542-402d-bc28-d4fdea2084f1","Type":"ContainerStarted","Data":"aec4f0ce04052cf8348d2bfb247164bbe326ae9e2400cf6f94cd2e2a0abd6a80"} Oct 30 05:30:38.698794 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:38.698747 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-d5q8v" podStartSLOduration=7.196001957 podStartE2EDuration="16.698735549s" totalImagesPullingTime="9.502733592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:38.686201047 +0000 UTC m=+16.632467933" watchObservedRunningTime="2026-10-30 05:30:38.698735549 +0000 UTC m=+16.645002434" Oct 30 05:30:39.633578 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:39.633367 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"bfb5b08a3eb2e41d42eaeb743f2d1b9a297845f0bf23c74e530d0c0d147565e4"} Oct 30 05:30:39.633913 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:39.633591 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"feb35c58654bffd70a1a5682b2731b3942234cf696f31f6644608e46d037af1d"} Oct 30 05:30:39.633913 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:39.633649 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"8f85769777867f47eb0bec206792b0ab054d9e1591b533cc69011b420bbce552"} Oct 30 05:30:39.633913 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:39.633682 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"10fde89057f9b3aa64904a951d3629bf5fab51511d5c893330914a3b988995e6"} Oct 30 05:30:39.633913 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:39.633712 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"19632baf3cce886c78841d07b6694a5450d5139b61c865126e5c4fbc1f60fb3a"} Oct 30 05:30:39.633913 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:39.633741 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"fec14a5b02bae2c070342886a8e3c4895d05306d25cae9b2163f0dfecbedbc4b"} Oct 30 05:30:40.172977 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.172947 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:40.173146 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.173010 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:40.173146 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.173069 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:40.173146 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.173094 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:40.173146 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.173120 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:30:56.173106369 +0000 UTC m=+34.119373254 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:40.173146 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.173139 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:30:56.173124639 +0000 UTC m=+34.119391509 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:40.274226 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.274200 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:40.274323 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.274299 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:40.274323 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.274311 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:40.274323 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.274320 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:40.274425 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.274356 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:30:56.274346244 +0000 UTC m=+34.220613134 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:40.457669 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.457596 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:40.457795 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.457690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:40.457795 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.457692 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:40.457795 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.457755 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:40.457795 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.457773 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:40.458058 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:40.457822 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:40.680876 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:40.680851 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wf8z_67e26d01-cd9d-4e17-bd57-f7eb78cdc92e/dns-node-resolver/0.log" Oct 30 05:30:41.618365 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:41.618339 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gbpks_42226e87-0da6-4fa1-8d6c-842368393e3e/node-ca/0.log" Oct 30 05:30:41.640740 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:41.640707 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"ee4432e8b47e770bc0a8bd07ff481f5844ac728c2be24bf14fe318f0e12f7c29"} Oct 30 05:30:42.458106 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:42.458080 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:42.458610 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:42.458146 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:42.458610 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:42.458174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:42.458610 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:42.458328 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:42.458610 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:42.458359 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:42.458610 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:42.458461 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:43.648351 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:43.648309 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" event={"ID":"93637adc-fd32-44b0-89ba-9a26152b5c79","Type":"ContainerStarted","Data":"1af0e0986cb9e17b6d6471f0a57dbdfe3610575f24faf5ebead7c136d0003ba0"} Oct 30 05:30:43.648925 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:43.648585 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:43.648925 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:43.648664 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:43.663108 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:43.663086 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:43.736663 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:43.736614 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" podStartSLOduration=7.418033111 podStartE2EDuration="21.736600259s" totalImagesPullingTime="14.318567148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:30:43.724087529 +0000 UTC m=+21.670354425" watchObservedRunningTime="2026-10-30 05:30:43.736600259 +0000 UTC m=+21.682867143" Oct 30 05:30:44.458185 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:44.458158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:44.458346 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:44.458213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:44.458346 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:44.458225 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:44.458346 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:44.458301 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:44.458440 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:44.458380 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:44.458473 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:44.458440 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:44.650346 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:44.650320 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:44.664657 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:44.664636 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:30:46.457547 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:46.457499 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:46.457547 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:46.457538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:46.458041 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:46.457611 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:46.458041 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:46.457648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:46.458041 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:46.457755 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:46.458041 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:46.457818 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:48.457714 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:48.457677 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:48.457714 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:48.457690 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:48.457714 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:48.457706 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:48.458280 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:48.457779 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:48.458280 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:48.457830 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:48.458280 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:48.457896 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:50.458008 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:50.457830 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:50.458444 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:50.457846 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:50.458444 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:50.458197 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:50.458444 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:50.458080 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:50.458444 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:50.457869 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:50.458444 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:50.458294 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:52.458845 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:52.458812 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:52.459303 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:52.458884 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:52.459303 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:52.458974 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:52.459303 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:52.459059 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:52.459303 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:52.459117 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:52.459303 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:52.459177 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:54.457586 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:54.457549 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:54.457586 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:54.457568 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:54.458078 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:54.457648 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:54.458078 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:54.457766 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:54.458078 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:54.457829 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:54.458078 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:54.457909 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:54.574633 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:54.574605 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:54.574787 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:54.574710 2581 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:30:54.575227 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:54.575203 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:30:56.189746 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:56.189710 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:56.190166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:56.189756 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:56.190166 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.189855 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:56.190166 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.189907 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:56.190166 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.189911 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret podName:2e463762-df03-4b5c-be5f-19892ed3d24b nodeName:}" failed. No retries permitted until 2026-10-30 05:31:28.189893132 +0000 UTC m=+66.136160009 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret") pod "global-pull-secret-syncer-rzb85" (UID: "2e463762-df03-4b5c-be5f-19892ed3d24b") : object "kube-system"/"original-pull-secret" not registered Oct 30 05:30:56.190166 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.189955 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs podName:5f9e88e1-4405-45ca-8fad-9fbe5c20d41f nodeName:}" failed. No retries permitted until 2026-10-30 05:31:28.189943817 +0000 UTC m=+66.136210687 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs") pod "network-metrics-daemon-rkb2j" (UID: "5f9e88e1-4405-45ca-8fad-9fbe5c20d41f") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 05:30:56.291034 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:56.291000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:56.291200 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.291113 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 05:30:56.291200 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.291128 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 05:30:56.291200 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.291139 2581 projected.go:196] Error preparing data for projected volume kube-api-access-b87mf for pod openshift-network-diagnostics/network-check-target-lqd5k: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:56.291200 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.291182 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf podName:d0ca187c-6933-4dd5-ada2-130897bea3ae nodeName:}" failed. No retries permitted until 2026-10-30 05:31:28.291171163 +0000 UTC m=+66.237438026 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-b87mf" (UniqueName: "kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf") pod "network-check-target-lqd5k" (UID: "d0ca187c-6933-4dd5-ada2-130897bea3ae") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 05:30:56.458046 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:56.457950 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:56.458204 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:56.458068 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:56.458204 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:56.458092 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:56.458204 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.458074 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:30:56.458204 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.458165 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:56.458356 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:56.458270 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:58.457768 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:58.457725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:30:58.457768 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:58.457751 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:30:58.458259 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:30:58.457841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:30:58.458259 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:58.457848 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:30:58.458259 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:58.457920 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:30:58.458259 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:30:58.458003 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:31:00.457419 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.457389 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:00.457821 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.457389 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:00.457821 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:00.457501 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:31:00.457821 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.457522 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:00.457821 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:00.457626 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:31:00.457821 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:00.457699 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:31:00.677833 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.677762 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lqd5k"] Oct 30 05:31:00.680805 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.680712 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzb85"] Oct 30 05:31:00.681702 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.681672 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkb2j"] Oct 30 05:31:00.683509 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.683483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:00.683644 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.683483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:00.683644 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:00.683483 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:00.683644 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:00.683605 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:31:00.683807 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:00.683679 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:31:00.683807 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:00.683767 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:31:02.459627 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:02.459592 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:02.460185 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:02.459636 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:02.460185 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:02.459702 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:02.460185 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:02.459734 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:31:02.460185 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:02.459788 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:31:02.460185 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:02.459828 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:31:04.458453 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:04.458418 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:04.458453 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:04.458443 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:04.458932 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:04.458640 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rkb2j" podUID="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" Oct 30 05:31:04.458932 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:04.458667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:04.458932 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:04.458760 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lqd5k" podUID="d0ca187c-6933-4dd5-ada2-130897bea3ae" Oct 30 05:31:04.458932 ip-10-0-129-175 kubenswrapper[2581]: E1030 05:31:04.458821 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rzb85" podUID="2e463762-df03-4b5c-be5f-19892ed3d24b" Oct 30 05:31:05.871601 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.871234 2581 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeReady" Oct 30 05:31:05.872069 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.871721 2581 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 05:31:05.944449 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.944413 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k22vf"] Oct 30 05:31:05.946371 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.946354 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:05.969453 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.969418 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p69qm"] Oct 30 05:31:05.971975 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.971912 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:05.991766 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.991734 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-gpq7r"] Oct 30 05:31:05.993816 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.993789 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-plfkr"] Oct 30 05:31:05.993940 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.993890 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:05.995691 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:05.995670 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.003630 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.003570 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:31:06.003630 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.003573 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:31:06.004026 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.003709 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:31:06.004026 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.003800 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2znb\"" Oct 30 05:31:06.004026 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.003896 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnf9b\"" Oct 30 05:31:06.004915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.004879 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p69qm"] Oct 30 05:31:06.021440 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.021407 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gpq7r"] Oct 30 05:31:06.026301 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.023030 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-plfkr"] Oct 30 05:31:06.043732 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.043694 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:31:06.043732 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.043703 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:31:06.043975 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.043956 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:31:06.044035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.043984 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:31:06.044035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.044001 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:31:06.044035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.044016 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:31:06.044035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.044020 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:31:06.044035 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.044001 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:31:06.044197 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.044051 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:31:06.044197 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.044055 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:31:06.060674 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060636 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-root\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.060823 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-metrics-client-ca\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.060823 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060711 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.060889 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060837 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85bc42bf-db08-410e-9eae-912afc399db4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.060889 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060879 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85bc42bf-db08-410e-9eae-912afc399db4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.060948 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060901 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-sys\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.060979 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060950 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-textfile\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.060979 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.060967 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-tls\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.061038 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.061023 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85bc42bf-db08-410e-9eae-912afc399db4-crio-socket\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.061070 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.061054 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pktl4\" (UniqueName: \"kubernetes.io/projected/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-kube-api-access-pktl4\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.061101 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.061087 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.061174 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.061156 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85bc42bf-db08-410e-9eae-912afc399db4-data-volume\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.061207 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.061186 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-wtmp\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.061241 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.061206 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sxt\" (UniqueName: \"kubernetes.io/projected/85bc42bf-db08-410e-9eae-912afc399db4-kube-api-access-t5sxt\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.096582 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.096519 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:31:06.096753 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.096660 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzzmx\"" Oct 30 05:31:06.096823 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.096810 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:31:06.096964 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.096953 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9q7xl\"" Oct 30 05:31:06.161895 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.161795 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2prr6\" (UniqueName: \"kubernetes.io/projected/54b31cad-aaef-4ade-adf9-4664d673e085-kube-api-access-2prr6\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.161895 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.161847 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85bc42bf-db08-410e-9eae-912afc399db4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.161895 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.161866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-sys\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.161911 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-textfile\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.161939 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-tls\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.161972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85bc42bf-db08-410e-9eae-912afc399db4-crio-socket\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162032 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-sys\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162077 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktl4\" (UniqueName: \"kubernetes.io/projected/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-kube-api-access-pktl4\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162125 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162166 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tj47h\" (UniqueName: \"kubernetes.io/projected/a1db5f90-a5e5-40c1-b500-353c07018ff3-kube-api-access-tj47h\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162180 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85bc42bf-db08-410e-9eae-912afc399db4-crio-socket\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85bc42bf-db08-410e-9eae-912afc399db4-data-volume\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162231 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-wtmp\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162259 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sxt\" (UniqueName: \"kubernetes.io/projected/85bc42bf-db08-410e-9eae-912afc399db4-kube-api-access-t5sxt\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-root\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162313 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-textfile\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162359 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-root\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162403 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b31cad-aaef-4ade-adf9-4664d673e085-tmp-dir\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162438 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-wtmp\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162493 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1db5f90-a5e5-40c1-b500-353c07018ff3-cert\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162504 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-metrics-client-ca\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162557 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162575 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85bc42bf-db08-410e-9eae-912afc399db4-data-volume\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54b31cad-aaef-4ade-adf9-4664d673e085-config-volume\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162671 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85bc42bf-db08-410e-9eae-912afc399db4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162713 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b31cad-aaef-4ade-adf9-4664d673e085-metrics-tls\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.162908 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.162780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85bc42bf-db08-410e-9eae-912afc399db4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.163140 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.163113 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-metrics-client-ca\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.163140 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.163122 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.166201 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.166173 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.166336 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.166182 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-tls\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.166336 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.166225 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85bc42bf-db08-410e-9eae-912afc399db4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.187753 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.187719 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sxt\" (UniqueName: \"kubernetes.io/projected/85bc42bf-db08-410e-9eae-912afc399db4-kube-api-access-t5sxt\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.187921 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.187788 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktl4\" (UniqueName: \"kubernetes.io/projected/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-kube-api-access-pktl4\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.259508 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.259462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:06.263190 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263166 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj47h\" (UniqueName: \"kubernetes.io/projected/a1db5f90-a5e5-40c1-b500-353c07018ff3-kube-api-access-tj47h\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.263276 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263205 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b31cad-aaef-4ade-adf9-4664d673e085-tmp-dir\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.263276 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263225 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1db5f90-a5e5-40c1-b500-353c07018ff3-cert\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.263276 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54b31cad-aaef-4ade-adf9-4664d673e085-config-volume\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.263276 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263266 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b31cad-aaef-4ade-adf9-4664d673e085-metrics-tls\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.263456 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263310 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prr6\" (UniqueName: \"kubernetes.io/projected/54b31cad-aaef-4ade-adf9-4664d673e085-kube-api-access-2prr6\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.263639 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263609 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b31cad-aaef-4ade-adf9-4664d673e085-tmp-dir\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.263903 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.263882 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54b31cad-aaef-4ade-adf9-4664d673e085-config-volume\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.265850 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.265821 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b31cad-aaef-4ade-adf9-4664d673e085-metrics-tls\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.266047 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.266023 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1db5f90-a5e5-40c1-b500-353c07018ff3-cert\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.286151 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.286114 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:06.288819 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.288787 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prr6\" (UniqueName: \"kubernetes.io/projected/54b31cad-aaef-4ade-adf9-4664d673e085-kube-api-access-2prr6\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.288973 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.288954 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj47h\" (UniqueName: \"kubernetes.io/projected/a1db5f90-a5e5-40c1-b500-353c07018ff3-kube-api-access-tj47h\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.309270 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.309234 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:06.315207 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.315181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:06.458046 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.457949 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:06.458046 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.457975 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:06.458268 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.458228 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:06.473625 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.473595 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:31:06.473796 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.473736 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:31:06.473796 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.473766 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\"" Oct 30 05:31:06.473899 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.473868 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:31:06.473947 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.473902 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\"" Oct 30 05:31:06.473992 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.473957 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\"" Oct 30 05:31:06.474103 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.474084 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:31:06.616461 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.616425 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-gpq7r"] Oct 30 05:31:06.620694 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.620664 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p69qm"] Oct 30 05:31:06.621165 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:06.621135 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1db5f90_a5e5_40c1_b500_353c07018ff3.slice/crio-96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70 WatchSource:0}: Error finding container 96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70: Status 404 returned error can't find the container with id 96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70 Oct 30 05:31:06.643776 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.643625 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-plfkr"] Oct 30 05:31:06.646828 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:06.646794 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54b31cad_aaef_4ade_adf9_4664d673e085.slice/crio-32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b WatchSource:0}: Error finding container 32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b: Status 404 returned error can't find the container with id 32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b Oct 30 05:31:06.698707 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.698671 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-plfkr" event={"ID":"54b31cad-aaef-4ade-adf9-4664d673e085","Type":"ContainerStarted","Data":"32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b"} Oct 30 05:31:06.699915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.699883 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gpq7r" event={"ID":"a1db5f90-a5e5-40c1-b500-353c07018ff3","Type":"ContainerStarted","Data":"96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70"} Oct 30 05:31:06.701141 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.701114 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p69qm" event={"ID":"85bc42bf-db08-410e-9eae-912afc399db4","Type":"ContainerStarted","Data":"65b4a76b7932511ece4fd3a0a2207015b2fec5d021d5877e9a0a081eb0fe8e99"} Oct 30 05:31:06.702268 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:06.702246 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k22vf" event={"ID":"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a","Type":"ContainerStarted","Data":"7e0e3f7c9585118f18cf13b64bc5185e11437c9c717aa6773a8624040a651ccd"} Oct 30 05:31:07.381370 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.381332 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:07.384775 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.384743 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.399825 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.399795 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:31:07.399963 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.399916 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:31:07.399963 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.399928 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:31:07.400080 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.399969 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:31:07.401016 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.400992 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:31:07.401149 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.401133 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:31:07.401269 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.401214 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:31:07.405458 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.405387 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:31:07.406509 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.406460 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lsrxx\"" Oct 30 05:31:07.436980 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.436939 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:31:07.438169 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.438144 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:07.473955 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.473919 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474056 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.473980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474170 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474063 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474170 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474290 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474342 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474318 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474394 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474439 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474439 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474424 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474568 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474568 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474568 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474490 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.474568 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.474557 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575682 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575648 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575682 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575715 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575732 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575753 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575800 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575831 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575850 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575866 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.575915 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.575910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.577595 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.577559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.579336 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.579309 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.579576 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.579554 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.582379 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.582331 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.583000 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.582952 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.583328 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.583249 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.583328 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.583287 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.583897 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.583877 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.584301 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.584269 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.584420 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.584401 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.585781 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.585755 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.588469 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.588440 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.606469 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.606408 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:07.707453 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.707366 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p69qm" event={"ID":"85bc42bf-db08-410e-9eae-912afc399db4","Type":"ContainerStarted","Data":"e2eaf86459b27f9bf7260c836137ff2054ce9b734694d8d126ae545bde40b2a1"} Oct 30 05:31:07.707453 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.707448 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p69qm" event={"ID":"85bc42bf-db08-410e-9eae-912afc399db4","Type":"ContainerStarted","Data":"b5fde0fdf556617d6d1c64131559c5efbb58fcdea346f86ab31c3e9b67c07d15"} Oct 30 05:31:07.709990 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.709910 2581 generic.go:356] "Generic (PLEG): container finished" podID="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" containerID="5a127c74a18d7fff5f2a7592baf6776c7932cec3f9e2ae50eca2362c95d49678" exitCode=0 Oct 30 05:31:07.710131 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.709985 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k22vf" event={"ID":"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a","Type":"ContainerDied","Data":"5a127c74a18d7fff5f2a7592baf6776c7932cec3f9e2ae50eca2362c95d49678"} Oct 30 05:31:07.745066 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:07.745028 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:08.109389 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.109352 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:08.122234 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.122202 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5b554f6f57-vkkxh"] Oct 30 05:31:08.126165 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.126023 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.127558 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:08.127513 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7473d6d4_13de_4ce5_acc3_771d24ca8b88.slice/crio-5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7 WatchSource:0}: Error finding container 5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7: Status 404 returned error can't find the container with id 5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7 Oct 30 05:31:08.144019 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.143986 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 05:31:08.144213 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.144089 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 05:31:08.144213 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.144137 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 05:31:08.144213 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.144147 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 05:31:08.144213 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.144170 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 05:31:08.144419 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.144229 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9l0qo440ric9d\"" Oct 30 05:31:08.144419 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.144363 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j88tp\"" Oct 30 05:31:08.191433 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.191397 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b554f6f57-vkkxh"] Oct 30 05:31:08.284603 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284549 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc069a86-752c-499c-98a0-c6ad926307f7-metrics-client-ca\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284603 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284607 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284637 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284665 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284741 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwp2\" (UniqueName: \"kubernetes.io/projected/cc069a86-752c-499c-98a0-c6ad926307f7-kube-api-access-vfwp2\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc069a86-752c-499c-98a0-c6ad926307f7-tmp\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284790 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.284842 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.284829 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-grpc-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386263 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386174 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwp2\" (UniqueName: \"kubernetes.io/projected/cc069a86-752c-499c-98a0-c6ad926307f7-kube-api-access-vfwp2\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386263 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc069a86-752c-499c-98a0-c6ad926307f7-tmp\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386280 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-grpc-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386403 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc069a86-752c-499c-98a0-c6ad926307f7-metrics-client-ca\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386490 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.386749 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.387022 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.386782 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc069a86-752c-499c-98a0-c6ad926307f7-tmp\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.387692 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.387568 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc069a86-752c-499c-98a0-c6ad926307f7-metrics-client-ca\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.389804 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.389758 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.390416 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.390397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.391464 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.391439 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-grpc-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.392557 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.392469 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.392895 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.392874 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.399463 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.399424 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.416843 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.416772 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwp2\" (UniqueName: \"kubernetes.io/projected/cc069a86-752c-499c-98a0-c6ad926307f7-kube-api-access-vfwp2\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.454369 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.454330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:08.726825 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.726724 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7"} Oct 30 05:31:08.731145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.731098 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k22vf" event={"ID":"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a","Type":"ContainerStarted","Data":"c82c81905ce45f775811fb4a9a0fe2aee05138b1e1533b1ac84d991f0c531ce5"} Oct 30 05:31:08.731285 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.731178 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-k22vf" event={"ID":"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a","Type":"ContainerStarted","Data":"15c2212e18b8229153aee9f843501e04da19fa0c2d2c17d447eea395fb6beea8"} Oct 30 05:31:08.777547 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.776332 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-k22vf" podStartSLOduration=2.586560572 podStartE2EDuration="3.77631136s" totalImagesPullingTime="1.09837026s" totalInitContainerRuntime="91.380528ms" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:08.77579104 +0000 UTC m=+46.722057926" watchObservedRunningTime="2026-10-30 05:31:08.77631136 +0000 UTC m=+46.722578246" Oct 30 05:31:08.802814 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:08.802782 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5b554f6f57-vkkxh"] Oct 30 05:31:08.804521 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:08.804482 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc069a86_752c_499c_98a0_c6ad926307f7.slice/crio-bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3 WatchSource:0}: Error finding container bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3: Status 404 returned error can't find the container with id bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3 Oct 30 05:31:09.737413 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.737372 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-plfkr" event={"ID":"54b31cad-aaef-4ade-adf9-4664d673e085","Type":"ContainerStarted","Data":"fcae8096f7faf2501f2a2cde69b7d897a1003afc8de3599217f656ca7306e63b"} Oct 30 05:31:09.737937 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.737441 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-plfkr" event={"ID":"54b31cad-aaef-4ade-adf9-4664d673e085","Type":"ContainerStarted","Data":"bd975be820223fa923204b94ddce5733e89c9104fcc3a9506c9dce9100333e00"} Oct 30 05:31:09.737937 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.737509 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:09.739765 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.739732 2581 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb" exitCode=0 Oct 30 05:31:09.739903 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.739775 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb"} Oct 30 05:31:09.742516 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.742493 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p69qm" event={"ID":"85bc42bf-db08-410e-9eae-912afc399db4","Type":"ContainerStarted","Data":"fba78ef0dc3cfb93d3371b3a3f55c2ddf91d0585d70f4da3ad46c11b59be3e37"} Oct 30 05:31:09.744193 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.744129 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3"} Oct 30 05:31:09.779677 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.779633 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-plfkr" podStartSLOduration=2.633912584 podStartE2EDuration="4.779618325s" totalImagesPullingTime="2.145705741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:09.779354064 +0000 UTC m=+47.725620951" watchObservedRunningTime="2026-10-30 05:31:09.779618325 +0000 UTC m=+47.725885188" Oct 30 05:31:09.824475 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:09.824419 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-p69qm" podStartSLOduration=2.497658411 podStartE2EDuration="4.824399348s" totalImagesPullingTime="2.326740937s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:09.823649661 +0000 UTC m=+47.769916546" watchObservedRunningTime="2026-10-30 05:31:09.824399348 +0000 UTC m=+47.770666237" Oct 30 05:31:10.394187 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.394154 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm"] Oct 30 05:31:10.397360 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.397326 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.412906 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.412878 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 05:31:10.415562 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.415518 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:31:10.416457 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.416430 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 05:31:10.416616 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.416548 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hj286\"" Oct 30 05:31:10.416698 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.416621 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 05:31:10.425430 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.425336 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6u3rkoovj0l3b\"" Oct 30 05:31:10.428975 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.428939 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm"] Oct 30 05:31:10.511681 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511641 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fdffe71b-09d7-4f44-9d2e-45cae3f24023-audit-log\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.511863 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511698 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.511863 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-tls\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.511863 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511835 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-metrics-server-audit-profiles\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.512034 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-client-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.512034 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511921 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr96b\" (UniqueName: \"kubernetes.io/projected/fdffe71b-09d7-4f44-9d2e-45cae3f24023-kube-api-access-mr96b\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.512034 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.511949 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.612950 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.612917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fdffe71b-09d7-4f44-9d2e-45cae3f24023-audit-log\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.612967 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.612994 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-tls\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.613027 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-metrics-server-audit-profiles\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.613054 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-client-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.613111 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr96b\" (UniqueName: \"kubernetes.io/projected/fdffe71b-09d7-4f44-9d2e-45cae3f24023-kube-api-access-mr96b\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613145 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.613142 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613430 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.613389 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fdffe71b-09d7-4f44-9d2e-45cae3f24023-audit-log\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.613911 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.613882 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.614371 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.614326 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-metrics-server-audit-profiles\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.616165 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.616139 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-client-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.616419 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.616377 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-tls\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.616601 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.616577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.639941 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.639905 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr96b\" (UniqueName: \"kubernetes.io/projected/fdffe71b-09d7-4f44-9d2e-45cae3f24023-kube-api-access-mr96b\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.711406 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.711327 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:10.735402 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.735366 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs"] Oct 30 05:31:10.744314 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.744287 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:10.750305 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.750276 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-gpq7r" event={"ID":"a1db5f90-a5e5-40c1-b500-353c07018ff3","Type":"ContainerStarted","Data":"d4e206bbd770de7ae51d429265293a13beefac3d44a92096d263e40b0bb1c967"} Oct 30 05:31:10.763672 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.760107 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs"] Oct 30 05:31:10.763672 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.760287 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 05:31:10.763672 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.760371 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w46sp\"" Oct 30 05:31:10.914848 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.914812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m2xr7\" (UniqueName: \"kubernetes.io/projected/72b235d4-33f4-48d7-946b-e218318547ca-kube-api-access-m2xr7\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:10.915017 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:10.914881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72b235d4-33f4-48d7-946b-e218318547ca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:11.015665 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.015634 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xr7\" (UniqueName: \"kubernetes.io/projected/72b235d4-33f4-48d7-946b-e218318547ca-kube-api-access-m2xr7\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:11.015794 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.015685 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72b235d4-33f4-48d7-946b-e218318547ca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:11.018371 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.018317 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72b235d4-33f4-48d7-946b-e218318547ca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:11.033224 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.032746 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-gpq7r" podStartSLOduration=2.8895334889999997 podStartE2EDuration="6.032732998s" totalImagesPullingTime="3.143199509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:10.833687242 +0000 UTC m=+48.779954128" watchObservedRunningTime="2026-10-30 05:31:11.032732998 +0000 UTC m=+48.978999882" Oct 30 05:31:11.033224 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.033177 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm"] Oct 30 05:31:11.039369 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.039319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xr7\" (UniqueName: \"kubernetes.io/projected/72b235d4-33f4-48d7-946b-e218318547ca-kube-api-access-m2xr7\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:11.039583 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:11.039555 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdffe71b_09d7_4f44_9d2e_45cae3f24023.slice/crio-6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03 WatchSource:0}: Error finding container 6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03: Status 404 returned error can't find the container with id 6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03 Oct 30 05:31:11.061592 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.061567 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:11.366822 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.366798 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs"] Oct 30 05:31:11.369390 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:11.369365 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72b235d4_33f4_48d7_946b_e218318547ca.slice/crio-704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881 WatchSource:0}: Error finding container 704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881: Status 404 returned error can't find the container with id 704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881 Oct 30 05:31:11.758629 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.758593 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a"} Oct 30 05:31:11.759098 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.758667 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c"} Oct 30 05:31:11.759098 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.758695 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc"} Oct 30 05:31:11.759098 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.758719 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0"} Oct 30 05:31:11.759098 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.758748 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1"} Oct 30 05:31:11.760228 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.760204 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" event={"ID":"fdffe71b-09d7-4f44-9d2e-45cae3f24023","Type":"ContainerStarted","Data":"6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03"} Oct 30 05:31:11.761721 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.761665 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" event={"ID":"72b235d4-33f4-48d7-946b-e218318547ca","Type":"ContainerStarted","Data":"704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881"} Oct 30 05:31:11.764775 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.764748 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"6ce8908569b3d50fde9a8d550a10796f5d616ec9f7662da64d88bc2a4a9ed971"} Oct 30 05:31:11.764878 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.764818 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"7dca5943784d52d183e9b6a222b9f09e247c8d9d257c302312bf01b07ef5f474"} Oct 30 05:31:11.764878 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:11.764849 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"0492801dd6a2f7454c879802679be0fe80b81de3bb30ce96422056adf63d3e7a"} Oct 30 05:31:12.773930 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.773836 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerStarted","Data":"d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc"} Oct 30 05:31:12.775607 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.775566 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" event={"ID":"fdffe71b-09d7-4f44-9d2e-45cae3f24023","Type":"ContainerStarted","Data":"c88696485aaed04231dac473dcafb08b76ad22447a187ce8655071826c768cb3"} Oct 30 05:31:12.778856 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.778833 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"21b5111f8d9ab83ed5353cfb931948372085775dcc69772bd5465690da8aafc8"} Oct 30 05:31:12.778945 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.778886 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"b0104476ea92ad8b5cced33351271bfe0d6001277ca725729b92467b94f38a66"} Oct 30 05:31:12.778945 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.778912 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" event={"ID":"cc069a86-752c-499c-98a0-c6ad926307f7","Type":"ContainerStarted","Data":"e3a6c8a05ee75e78f3364553ca9bbbcaad049e74b34af43756de4f6f1b48811d"} Oct 30 05:31:12.779384 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.779369 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:12.806003 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.805974 2581 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:12.809785 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.809763 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.835788 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.835761 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:31:12.836290 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836266 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:31:12.836568 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836555 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:31:12.836650 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836636 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:31:12.836709 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836659 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:31:12.836759 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836704 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6w7wj\"" Oct 30 05:31:12.836759 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836717 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:31:12.836845 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.836825 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:31:12.846904 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.846887 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:31:12.846989 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.846925 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:31:12.846989 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.846904 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:31:12.860728 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.860711 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:31:12.860922 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.860904 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:31:12.861049 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.860946 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:31:12.861049 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.860951 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6avl1qpif1ntl\"" Oct 30 05:31:12.867267 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.863129 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:31:12.876487 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.876444 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.731682479 podStartE2EDuration="5.876428611s" totalImagesPullingTime="3.129980709s" totalInitContainerRuntime="14.765423ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:12.876162963 +0000 UTC m=+50.822429847" watchObservedRunningTime="2026-10-30 05:31:12.876428611 +0000 UTC m=+50.822695498" Oct 30 05:31:12.881444 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.881425 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:12.932988 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.932948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.932988 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.932985 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.933205 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.933013 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.933205 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.933037 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.933205 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.933157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.933205 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.933180 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.933873 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.933846 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934135 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.933890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934231 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934212 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934284 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934242 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934333 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934297 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934333 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934313 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934426 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934341 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934426 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934371 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.934426 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934386 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.936128 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934695 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.936128 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.934733 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.936128 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.935284 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.936128 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.935367 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.936128 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.935439 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:12.938009 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.937962 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" podStartSLOduration=2.122922469 podStartE2EDuration="4.937945191s" totalImagesPullingTime="2.815022722s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:12.936500356 +0000 UTC m=+50.882767244" watchObservedRunningTime="2026-10-30 05:31:12.937945191 +0000 UTC m=+50.884212077" Oct 30 05:31:12.979876 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:12.979829 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" podStartSLOduration=1.53799822 podStartE2EDuration="2.97981507s" totalImagesPullingTime="1.44181685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:12.978657416 +0000 UTC m=+50.924924314" watchObservedRunningTime="2026-10-30 05:31:12.97981507 +0000 UTC m=+50.926081954" Oct 30 05:31:13.036292 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036292 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036292 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036260 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036292 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036290 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036314 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036335 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036376 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036399 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036448 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036467 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036492 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036593 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.036613 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036609 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037147 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037147 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037147 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037147 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036718 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037147 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037147 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.036745 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.037442 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.037328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.038096 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.038031 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.038275 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.038247 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.038667 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.038637 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.039227 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.039199 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.039789 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.039762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.040000 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.039963 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.040636 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.040614 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.041204 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.041175 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.041386 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.041355 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.041629 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.041609 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.041779 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.041758 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.042378 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.041985 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.042378 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.042224 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.042378 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.042306 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.042378 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.042353 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.042731 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.042714 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.043470 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.043453 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.064994 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.064970 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.135657 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.135626 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:13.462871 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.462846 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:13.465255 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:13.465232 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cac2a8_351c_4736_8732_15cc5af686a7.slice/crio-bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2 WatchSource:0}: Error finding container bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2: Status 404 returned error can't find the container with id bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2 Oct 30 05:31:13.783621 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.783583 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" event={"ID":"72b235d4-33f4-48d7-946b-e218318547ca","Type":"ContainerStarted","Data":"59d652e17da37c166e7ae7af2e15217c2dac9a42cf5d92d86f3ec6d50da7ca69"} Oct 30 05:31:13.784053 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.783743 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:13.785270 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.785241 2581 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" exitCode=0 Oct 30 05:31:13.785412 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.785276 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} Oct 30 05:31:13.785412 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.785336 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2"} Oct 30 05:31:13.789129 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.789113 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:13.823379 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:13.823337 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" podStartSLOduration=2.338205205 podStartE2EDuration="3.823324616s" totalImagesPullingTime="1.485119411s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:13.822979105 +0000 UTC m=+51.769245991" watchObservedRunningTime="2026-10-30 05:31:13.823324616 +0000 UTC m=+51.769591530" Oct 30 05:31:16.667410 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:16.667376 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:17.808833 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.808791 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} Oct 30 05:31:17.809243 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.808891 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} Oct 30 05:31:17.809243 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.808917 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} Oct 30 05:31:17.809243 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.808947 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} Oct 30 05:31:17.809243 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.808973 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} Oct 30 05:31:17.809243 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.808998 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerStarted","Data":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} Oct 30 05:31:17.865694 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:17.865636 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.840370289 podStartE2EDuration="5.865620132s" totalImagesPullingTime="3.008981153s" totalInitContainerRuntime="16.26869ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:17.864356002 +0000 UTC m=+55.810622889" watchObservedRunningTime="2026-10-30 05:31:17.865620132 +0000 UTC m=+55.811887017" Oct 30 05:31:18.136119 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:18.136024 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:19.753404 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:19.753372 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:19.795254 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:19.795227 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:28.281681 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.281644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:28.282106 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.281705 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:28.295634 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.295615 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:31:28.295683 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.295641 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:31:28.304896 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.304866 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:28.304896 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.304881 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:28.382604 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.382580 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:28.389925 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.389911 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\"" Oct 30 05:31:28.396063 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.396049 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\"" Oct 30 05:31:28.396122 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.396109 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:31:28.397167 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.397154 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:28.403367 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.403003 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:28.416934 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.416914 2581 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:31:28.426275 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.426254 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:28.683550 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.683473 2581 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\"" Oct 30 05:31:28.690661 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.690641 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:28.710574 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.710549 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rkb2j"] Oct 30 05:31:28.713247 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:28.713221 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f9e88e1_4405_45ca_8fad_9fbe5c20d41f.slice/crio-885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c WatchSource:0}: Error finding container 885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c: Status 404 returned error can't find the container with id 885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c Oct 30 05:31:28.721816 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.721788 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rzb85"] Oct 30 05:31:28.846856 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.846824 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkb2j" event={"ID":"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f","Type":"ContainerStarted","Data":"885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c"} Oct 30 05:31:28.847951 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.847926 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rzb85" event={"ID":"2e463762-df03-4b5c-be5f-19892ed3d24b","Type":"ContainerStarted","Data":"04f43f272ee8921e1b4e62bce809a2bb412e1d1e16eefc96e2c7f8b14dc73bee"} Oct 30 05:31:28.993793 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:28.993771 2581 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lqd5k"] Oct 30 05:31:28.995787 ip-10-0-129-175 kubenswrapper[2581]: W1030 05:31:28.995761 2581 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ca187c_6933_4dd5_ada2_130897bea3ae.slice/crio-5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65 WatchSource:0}: Error finding container 5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65: Status 404 returned error can't find the container with id 5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65 Oct 30 05:31:29.857662 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:29.857631 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkb2j" event={"ID":"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f","Type":"ContainerStarted","Data":"d67670a3b7ade6739e084c73788beda1076b09ff762c579b1ebd776ded71885a"} Oct 30 05:31:29.859902 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:29.859871 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lqd5k" event={"ID":"d0ca187c-6933-4dd5-ada2-130897bea3ae","Type":"ContainerStarted","Data":"5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65"} Oct 30 05:31:30.712200 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:30.712159 2581 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:30.712395 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:30.712238 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:30.867027 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:30.866993 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rkb2j" event={"ID":"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f","Type":"ContainerStarted","Data":"6e040c1b0f561efb53c7b7a188867c35573e957539d454623eb7e294bcb7c664"} Oct 30 05:31:30.908010 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:30.907963 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rkb2j" podStartSLOduration=67.903130356 podStartE2EDuration="1m8.907921556s" totalImagesPullingTime="1.0047912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:30.907557387 +0000 UTC m=+68.853824274" watchObservedRunningTime="2026-10-30 05:31:30.907921556 +0000 UTC m=+68.854188444" Oct 30 05:31:32.877712 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:32.877674 2581 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lqd5k" event={"ID":"d0ca187c-6933-4dd5-ada2-130897bea3ae","Type":"ContainerStarted","Data":"d7e16974d99e6705b18d91d3fe1bb68490ac36925f5c10c8e1b054989185e1d6"} Oct 30 05:31:32.878199 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:32.877779 2581 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:32.917541 ip-10-0-129-175 kubenswrapper[2581]: I1030 05:31:32.917483 2581 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lqd5k" podStartSLOduration=67.922982698 podStartE2EDuration="1m10.917466452s" totalImagesPullingTime="2.994483754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:30:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:32.916878157 +0000 UTC m=+70.863145050" watchObservedRunningTime="2026-10-30 05:31:32.917466452 +0000 UTC m=+70.863733379" Oct 30 05:31:33.356058 ip-10-0-129-175 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 05:31:33.368897 ip-10-0-129-175 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 05:31:33.369068 ip-10-0-129-175 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 05:31:33.369136 ip-10-0-129-175 systemd[1]: kubelet.service: Consumed 4.178s CPU time. Oct 30 05:31:33.384182 ip-10-0-129-175 systemd[1]: Starting Kubernetes Kubelet... Oct 30 05:31:33.456873 ip-10-0-129-175 kubenswrapper[5565]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:33.456873 ip-10-0-129-175 kubenswrapper[5565]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 05:31:33.456873 ip-10-0-129-175 kubenswrapper[5565]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:33.456873 ip-10-0-129-175 kubenswrapper[5565]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 05:31:33.459148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459036 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.459148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459148 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.459148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459153 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459156 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459159 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459162 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459165 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459167 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459170 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459173 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459176 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459180 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459183 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459186 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459189 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459192 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459195 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459198 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459201 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459203 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459206 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459210 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.459256 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459212 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459215 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459218 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459221 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459223 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459233 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459236 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459239 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459242 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459245 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459248 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459251 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459254 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459256 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459259 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459262 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459265 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459268 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459271 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459274 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.459829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459277 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459280 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459283 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459285 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459288 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459290 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459293 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459296 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459298 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459301 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459303 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459306 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459308 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459312 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459315 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459318 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459323 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459329 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459332 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.460342 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459336 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459351 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459354 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459357 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459360 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459362 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459365 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459368 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459371 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459375 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459377 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459380 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459383 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459385 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459388 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459392 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459396 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459399 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459402 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.460953 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459405 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459408 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459411 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459413 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459416 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459419 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459421 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459424 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459427 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459430 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459433 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459435 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459438 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459441 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459446 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459449 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459453 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459455 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459458 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.461441 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459461 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459464 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459467 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459469 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459472 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459474 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.459479 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460440 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460450 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460454 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460457 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460460 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460463 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460466 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460470 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460473 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460475 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460478 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460481 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.461936 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460484 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460487 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460489 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460492 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460495 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460497 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460500 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460504 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460509 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460512 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460515 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460517 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460520 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460538 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460543 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460547 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460551 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460555 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460558 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460563 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.462431 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460566 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460570 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460575 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460578 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460581 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460583 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460587 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460590 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460593 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460595 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460598 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460602 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460604 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460607 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460611 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460613 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460616 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460619 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460621 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460624 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.462994 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460636 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460639 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460642 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460644 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460647 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460650 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460652 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460655 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460658 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460660 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460663 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460666 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460669 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460672 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460674 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460677 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460680 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460682 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460687 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.463544 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460692 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460695 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460698 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460701 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460704 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460707 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460710 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460713 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460715 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460718 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460721 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460724 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460726 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460731 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460734 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460737 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460740 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460742 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460745 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460748 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.464072 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460751 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460754 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460756 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460759 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460764 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460767 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460771 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460775 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460778 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460781 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460784 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460786 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460789 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460792 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.460794 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461349 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461355 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461358 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461361 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.464635 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461364 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461367 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461370 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461374 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461378 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461381 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461387 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461391 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461394 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461397 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461400 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461402 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461405 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461408 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461411 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461414 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461416 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461419 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461422 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.465114 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461425 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461428 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461431 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461433 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461436 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461441 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461444 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461447 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461450 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461452 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461455 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461458 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461461 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461463 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461466 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461469 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461471 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461475 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461480 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461484 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.465645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461487 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461490 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461493 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461496 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461498 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461501 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461504 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461506 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461509 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461511 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461514 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461517 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461520 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461522 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461539 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461542 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461545 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461549 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461552 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461555 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.466148 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461558 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461560 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461563 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461566 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461568 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461571 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461573 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461576 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461579 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461582 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461585 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461588 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461591 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461594 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461598 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461601 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461604 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461606 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461609 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.466923 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461612 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461615 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461617 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461620 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461622 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461625 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461628 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461630 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461633 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461635 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461641 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461644 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461647 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461649 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461652 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461654 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461657 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461660 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461662 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.467716 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461665 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461668 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461670 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461673 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.461675 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461780 5565 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461796 5565 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461803 5565 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461809 5565 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461815 5565 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461819 5565 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461823 5565 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461833 5565 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461836 5565 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461839 5565 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461843 5565 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461846 5565 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461850 5565 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461853 5565 flags.go:64] FLAG: --cgroup-root="" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461856 5565 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461859 5565 flags.go:64] FLAG: --client-ca-file="" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461862 5565 flags.go:64] FLAG: --cloud-provider="external" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461866 5565 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461873 5565 flags.go:64] FLAG: --cluster-domain="" Oct 30 05:31:33.468639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461876 5565 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461880 5565 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461883 5565 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461887 5565 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461891 5565 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461895 5565 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461898 5565 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461902 5565 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461906 5565 flags.go:64] FLAG: --contention-profiling="false" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461909 5565 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461912 5565 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461916 5565 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461919 5565 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461923 5565 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461927 5565 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461930 5565 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461933 5565 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461936 5565 flags.go:64] FLAG: --enable-server="true" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461941 5565 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461946 5565 flags.go:64] FLAG: --event-burst="100" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461949 5565 flags.go:64] FLAG: --event-qps="50" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461952 5565 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461956 5565 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461959 5565 flags.go:64] FLAG: --eviction-hard="" Oct 30 05:31:33.469771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461964 5565 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461967 5565 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461970 5565 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461974 5565 flags.go:64] FLAG: --eviction-soft="" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461977 5565 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461980 5565 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461984 5565 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461988 5565 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461991 5565 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461994 5565 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.461997 5565 flags.go:64] FLAG: --feature-gates="" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462002 5565 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462005 5565 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462009 5565 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462012 5565 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462017 5565 flags.go:64] FLAG: --healthz-port="10248" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462021 5565 flags.go:64] FLAG: --help="false" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462024 5565 flags.go:64] FLAG: --hostname-override="ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462028 5565 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462031 5565 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462034 5565 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462038 5565 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462042 5565 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462045 5565 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 05:31:33.470884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462048 5565 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462051 5565 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462054 5565 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462059 5565 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462062 5565 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462065 5565 flags.go:64] FLAG: --kube-reserved="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462068 5565 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462071 5565 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462075 5565 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462078 5565 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462081 5565 flags.go:64] FLAG: --lock-file="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462084 5565 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462088 5565 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462091 5565 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462097 5565 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462102 5565 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462105 5565 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462108 5565 flags.go:64] FLAG: --logging-format="text" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462111 5565 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462115 5565 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462118 5565 flags.go:64] FLAG: --manifest-url="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462121 5565 flags.go:64] FLAG: --manifest-url-header="" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462126 5565 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462131 5565 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462135 5565 flags.go:64] FLAG: --max-pods="110" Oct 30 05:31:33.472077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462138 5565 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462142 5565 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462145 5565 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462149 5565 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462152 5565 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462155 5565 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462159 5565 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462167 5565 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462171 5565 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462174 5565 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462177 5565 flags.go:64] FLAG: --pod-cidr="" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462181 5565 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462184 5565 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462187 5565 flags.go:64] FLAG: --pods-per-core="0" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462191 5565 flags.go:64] FLAG: --port="10250" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462194 5565 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462197 5565 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b2035c58548f5275" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462201 5565 flags.go:64] FLAG: --qos-reserved="" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462204 5565 flags.go:64] FLAG: --read-only-port="10255" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462207 5565 flags.go:64] FLAG: --register-node="true" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462210 5565 flags.go:64] FLAG: --register-with-taints="" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462214 5565 flags.go:64] FLAG: --registry-burst="10" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462219 5565 flags.go:64] FLAG: --registry-qps="5" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462222 5565 flags.go:64] FLAG: --reserved-cpus="" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462225 5565 flags.go:64] FLAG: --reserved-memory="" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462229 5565 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 05:31:33.473177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462232 5565 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462235 5565 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462238 5565 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462241 5565 flags.go:64] FLAG: --runonce="false" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462246 5565 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462250 5565 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462253 5565 flags.go:64] FLAG: --seccomp-default="false" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462256 5565 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462260 5565 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462263 5565 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462266 5565 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462270 5565 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462273 5565 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462276 5565 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462279 5565 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462282 5565 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462285 5565 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462289 5565 flags.go:64] FLAG: --system-cgroups="" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462292 5565 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462299 5565 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462302 5565 flags.go:64] FLAG: --tls-cert-file="" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462305 5565 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462310 5565 flags.go:64] FLAG: --tls-min-version="" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462313 5565 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462317 5565 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 05:31:33.474351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462320 5565 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462322 5565 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462326 5565 flags.go:64] FLAG: --v="2" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462332 5565 flags.go:64] FLAG: --version="false" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462337 5565 flags.go:64] FLAG: --vmodule="" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462342 5565 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462345 5565 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462501 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462505 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462509 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462512 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462516 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462519 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462522 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462543 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462548 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462552 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462556 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462561 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462565 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462569 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.475443 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462573 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462577 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462581 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462586 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462590 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462595 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462599 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462603 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462608 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462612 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462616 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462620 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462624 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462630 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462637 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462643 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462647 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462652 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462657 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.476377 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462661 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462665 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462668 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462673 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462677 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462680 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462682 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462685 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462688 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462691 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462693 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462696 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462699 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462702 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462704 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462707 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462710 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462713 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462716 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462719 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.477208 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462722 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462725 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462728 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462731 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462734 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462737 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462739 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462744 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462746 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462749 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462752 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462755 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462758 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462761 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462763 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462767 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462770 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462772 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462775 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462778 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.477894 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462781 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462784 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462786 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462789 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462792 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462795 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462797 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462800 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462803 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462806 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462809 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462812 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462815 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462818 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462821 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462823 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462826 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462829 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462831 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.478464 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462835 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462838 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462841 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462844 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462847 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462850 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462853 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462855 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462859 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462862 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462865 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462870 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462873 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462876 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.462912 5565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462952 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.479044 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462958 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462960 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462964 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462967 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462970 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462973 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462976 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462979 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462981 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462984 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462987 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462990 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462993 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.462997 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463000 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463002 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463006 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463009 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463012 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463015 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.479762 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463018 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463021 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463024 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463026 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463030 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463033 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463035 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463038 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463041 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463044 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463046 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463049 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463051 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463054 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463058 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463062 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463065 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463068 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463087 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.480284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463091 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463094 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463097 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463099 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463105 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463108 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463111 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463114 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463116 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463119 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463122 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463124 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463128 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463131 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463133 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463136 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463139 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463143 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463146 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.480813 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463148 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463151 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463154 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463157 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463160 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463162 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463166 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463170 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463173 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463176 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463179 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463182 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463184 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463187 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463190 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463193 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463198 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463200 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463204 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463207 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.481499 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463209 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463212 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463214 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463217 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463220 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463223 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463225 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463228 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463231 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463233 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463238 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463240 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463243 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463246 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463248 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463251 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463254 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463257 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463260 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463262 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.482319 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463265 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463269 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463271 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463274 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463277 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463280 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.463283 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.469301 5565 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.469340 5565 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.469373 5565 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.469382 5565 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469506 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469517 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469523 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469545 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469550 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469555 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469563 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469572 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.482943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469578 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469584 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469589 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469594 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469599 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469605 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469610 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469616 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469621 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469625 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469630 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469634 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469639 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469645 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469651 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469656 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469661 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469666 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469670 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469676 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.483444 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469680 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469686 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469691 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469697 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469702 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469707 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469711 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469715 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469719 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469724 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469728 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469733 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469738 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469744 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469749 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469753 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469758 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469762 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469767 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469773 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.484016 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469778 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469782 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469786 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469791 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469796 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469800 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469804 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469809 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469814 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469819 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469824 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469829 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469834 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469840 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469845 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469851 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469855 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469860 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469865 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469870 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.484540 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469875 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469880 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469885 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469890 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469894 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469899 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469904 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469909 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469914 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469919 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469923 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469931 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469936 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469940 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469948 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469955 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469961 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469967 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469973 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.485052 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469977 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469981 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469987 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469992 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.469997 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470002 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470007 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470012 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470016 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470021 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470025 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470030 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470035 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470040 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470045 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470050 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470055 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470059 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.485573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470064 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.470139 5565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470403 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470414 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470419 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470424 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470429 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470435 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470440 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470445 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470450 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470455 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470460 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470464 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470469 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470474 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.486042 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470479 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470484 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470488 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470492 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470497 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470501 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470507 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470511 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470516 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470521 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470542 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470548 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470569 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470574 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470579 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470584 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470589 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470594 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470601 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.486563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470608 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470613 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470618 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470622 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470626 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470633 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470638 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470642 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470646 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470651 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470657 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470661 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470666 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470671 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470677 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470682 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470688 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470694 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470709 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.487216 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470716 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470722 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470727 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470733 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470737 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470741 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470747 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470752 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470757 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470761 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470766 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470771 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470775 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470780 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470784 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470788 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470793 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470797 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470802 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470811 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.487834 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470815 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470820 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470825 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470829 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470834 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470838 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470843 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470849 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470854 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470859 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470864 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470871 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470876 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470880 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470885 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470890 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470894 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470898 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470902 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470907 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.488423 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470911 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470915 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470920 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470924 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470929 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470933 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470939 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470944 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470949 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470953 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470958 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470963 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470970 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.470978 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.471034 5565 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471095 5565 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 05:31:33.488958 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471104 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471110 5565 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471116 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471120 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471125 5565 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471130 5565 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471134 5565 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471138 5565 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471143 5565 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471148 5565 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471153 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471157 5565 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471162 5565 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471167 5565 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471171 5565 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471175 5565 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471180 5565 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471184 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471189 5565 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471193 5565 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 05:31:33.489543 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471198 5565 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471202 5565 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471207 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471211 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471216 5565 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471221 5565 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471225 5565 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471230 5565 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471234 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471244 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471249 5565 feature_gate.go:386] unrecognized feature gate: Example Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471253 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471258 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471262 5565 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471266 5565 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471271 5565 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471275 5565 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471280 5565 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471284 5565 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471288 5565 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 05:31:33.490134 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471293 5565 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471298 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471302 5565 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471307 5565 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471311 5565 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471316 5565 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471320 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471325 5565 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471330 5565 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471335 5565 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471339 5565 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471344 5565 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471350 5565 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471355 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471359 5565 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471364 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471368 5565 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471372 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471377 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 05:31:33.490931 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471382 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471386 5565 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471399 5565 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471405 5565 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471410 5565 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471414 5565 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471419 5565 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471424 5565 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471428 5565 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471433 5565 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471440 5565 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471446 5565 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471451 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471456 5565 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471460 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471465 5565 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471470 5565 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471475 5565 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471480 5565 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 05:31:33.491485 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471484 5565 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471489 5565 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471493 5565 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471497 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471502 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471508 5565 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471512 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471517 5565 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471522 5565 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471545 5565 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471550 5565 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471554 5565 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471559 5565 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471564 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471568 5565 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471575 5565 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471579 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471584 5565 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471593 5565 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 05:31:33.492033 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471598 5565 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471602 5565 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471607 5565 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471612 5565 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471617 5565 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471621 5565 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471625 5565 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:33.471631 5565 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.471773 5565 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.473703 5565 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.473802 5565 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.474141 5565 server.go:1014] "Starting client certificate rotation" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.474282 5565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.474370 5565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 05:25:22 +0000 UTC" deadline="2028-05-06 06:26:01.089392384 +0000 UTC" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.474436 5565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13296h54m27.614959102s" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.475028 5565 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:31:33.492748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.477115 5565 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 05:31:33.493387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.478697 5565 log.go:25] "Validated CRI v1 runtime API" Oct 30 05:31:33.493387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.481062 5565 log.go:25] "Validated CRI v1 image API" Oct 30 05:31:33.493387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.482017 5565 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 05:31:33.493387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.485187 5565 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 f275cab1-49d4-415b-9415-000370bd54ab:/dev/nvme0n1p4 faf793b3-b677-47a8-beb4-fffcd7509d42:/dev/nvme0n1p3] Oct 30 05:31:33.493599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.485221 5565 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/04f43f272ee8921e1b4e62bce809a2bb412e1d1e16eefc96e2c7f8b14dc73bee/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/04f43f272ee8921e1b4e62bce809a2bb412e1d1e16eefc96e2c7f8b14dc73bee/userdata/shm major:0 minor:429 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65/userdata/shm major:0 minor:434 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/65b4a76b7932511ece4fd3a0a2207015b2fec5d021d5877e9a0a081eb0fe8e99/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/65b4a76b7932511ece4fd3a0a2207015b2fec5d021d5877e9a0a081eb0fe8e99/userdata/shm major:0 minor:212 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03/userdata/shm major:0 minor:296 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881/userdata/shm major:0 minor:319 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e0e3f7c9585118f18cf13b64bc5185e11437c9c717aa6773a8624040a651ccd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e0e3f7c9585118f18cf13b64bc5185e11437c9c717aa6773a8624040a651ccd/userdata/shm major:0 minor:206 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c/userdata/shm major:0 minor:430 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2/userdata/shm major:0 minor:392 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3/userdata/shm major:0 minor:266 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/09b60a4b-f264-48b1-a862-f5a54b7316da/volumes/kubernetes.io~projected/kube-api-access-bmjc5:{mountpoint:/var/lib/kubelet/pods/09b60a4b-f264-48b1-a862-f5a54b7316da/volumes/kubernetes.io~projected/kube-api-access-bmjc5 major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/config-reloader/2:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/config-reloader/2 major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/init-config-reloader/2:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/init-config-reloader/2 major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/prometheus/15:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/prometheus/15 major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~empty-dir/config-out major:0 minor:389 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~projected/kube-api-access-g9t9s:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~projected/kube-api-access-g9t9s major:0 minor:391 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~projected/tls-assets major:0 minor:382 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/config:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/config major:0 minor:387 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy major:0 minor:388 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-metrics-client-certs:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-metrics-client-certs major:0 minor:390 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web major:0 minor:383 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls major:0 minor:376 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls major:0 minor:384 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/web-config major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a/volumes/kubernetes.io~projected/kube-api-access-xn6rx:{mountpoint:/var/lib/kubelet/pods/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a/volumes/kubernetes.io~projected/kube-api-access-xn6rx major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2e463762-df03-4b5c-be5f-19892ed3d24b/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/2e463762-df03-4b5c-be5f-19892ed3d24b/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/42226e87-0da6-4fa1-8d6c-842368393e3e/volumes/kubernetes.io~projected/kube-api-access-r482q:{mountpoint:/var/lib/kubelet/pods/42226e87-0da6-4fa1-8d6c-842368393e3e/volumes/kubernetes.io~projected/kube-api-access-r482q major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d0513be-2542-402d-bc28-d4fdea2084f1/volumes/kubernetes.io~projected/kube-api-access-9dns2:{mountpoint:/var/lib/kubelet/pods/4d0513be-2542-402d-bc28-d4fdea2084f1/volumes/kubernetes.io~projected/kube-api-access-9dns2 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b31cad-aaef-4ade-adf9-4664d673e085/volumes/kubernetes.io~projected/kube-api-access-2prr6:{mountpoint:/var/lib/kubelet/pods/54b31cad-aaef-4ade-adf9-4664d673e085/volumes/kubernetes.io~projected/kube-api-access-2prr6 major:0 minor:210 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54b31cad-aaef-4ade-adf9-4664d673e085/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/54b31cad-aaef-4ade-adf9-4664d673e085/volumes/kubernetes.io~secret/metrics-tls major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f/volumes/kubernetes.io~projected/kube-api-access-4k454:{mountpoint:/var/lib/kubelet/pods/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f/volumes/kubernetes.io~projected/kube-api-access-4k454 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f/volumes/kubernetes.io~secret/metrics-certs major:0 minor:428 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e/volumes/kubernetes.io~projected/kube-api-access-glnts:{mountpoint:/var/lib/kubelet/pods/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e/volumes/kubernetes.io~projected/kube-api-access-glnts major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72b235d4-33f4-48d7-946b-e218318547ca/volumes/kubernetes.io~projected/kube-api-access-m2xr7:{mountpoint:/var/lib/kubelet/pods/72b235d4-33f4-48d7-946b-e218318547ca/volumes/kubernetes.io~projected/kube-api-access-m2xr7 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72b235d4-33f4-48d7-946b-e218318547ca/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/72b235d4-33f4-48d7-946b-e218318547ca/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/alertmanager/9 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/config-reloader/6 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~empty-dir/config-out major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~projected/kube-api-access-t9f99:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~projected/kube-api-access-t9f99 major:0 minor:233 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~projected/tls-assets major:0 minor:244 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/config-volume major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/web-config major:0 minor:235 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bc42bf-db08-410e-9eae-912afc399db4/volumes/kubernetes.io~projected/kube-api-access-t5sxt:{mountpoint:/var/lib/kubelet/pods/85bc42bf-db08-410e-9eae-912afc399db4/volumes/kubernetes.io~projected/kube-api-access-t5sxt major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/85bc42bf-db08-410e-9eae-912afc399db4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/85bc42bf-db08-410e-9eae-912afc399db4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volumes/kubernetes.io~projected/kube-api-access-pw8f6:{mountpoint:/var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volumes/kubernetes.io~projected/kube-api-access-pw8f6 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1db5f90-a5e5-40c1-b500-353c07018ff3/volumes/kubernetes.io~projected/kube-api-access-tj47h:{mountpoint:/var/lib/kubelet/pods/a1db5f90-a5e5-40c1-b500-353c07018ff3/volumes/kubernetes.io~projected/kube-api-access-tj47h major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1db5f90-a5e5-40c1-b500-353c07018ff3/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a1db5f90-a5e5-40c1-b500-353c07018ff3/volumes/kubernetes.io~secret/cert major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~projected/kube-api-access-pktl4:{mountpoint:/var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~projected/kube-api-access-pktl4 major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c6b480bd-5b3f-494c-a6e8-b59b15ba4818/volumes/kubernetes.io~projected/kube-api-access-lk7s6:{mountpoint:/var/lib/kubelet/pods/c6b480bd-5b3f-494c-a6e8-b59b15ba4818/volumes/kubernetes.io~projected/kube-api-access-lk7s6 major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~projected/kube-api-access-vfwp2:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~projected/kube-api-access-vfwp2 major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-grpc-tls:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-grpc-tls major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-tls:{mountpoint:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-tls major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d0ca187c-6933-4dd5-ada2-130897bea3ae/volumes/kubernetes.io~projected/kube-api-access-b87mf:{mountpoint:/var/lib/kubelet/pods/d0ca187c-6933-4dd5-ada2-130897bea3ae/volumes/kubernetes.io~projected/kube-api-access-b87mf major:0 minor:433 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~empty-dir/tmp major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~projected/kube-api-access-nns62:{mountpoint:/var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~projected/kube-api-access-nns62 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/eddd45fc-666b-4ba0-a049-10623018ea60/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/eddd45fc-666b-4ba0-a049-10623018ea60/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~projected/kube-api-access-mr96b:{mountpoint:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~projected/kube-api-access-mr96b major:0 minor:295 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:293 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:294 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:288 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/68ff5a6509ff902cd3dce12d664a45131f22c9001686cdc015721fd7c578e816/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/94aa409d14f7474cba9f8b6cf7ed88368a5ad190378f4b497c5ba0ee170617b3/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/847be4b137183e4b9e9c0679aa3b3dcb43e894242e5cb871c25c9e1f7041df60/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/55ccb575c376ce22d399002a6b4805b9f30ba4195a24a7de6fec1a303ded0975/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/b0b4cd37776de467a6aa57d2c2664aca1a0c4c9a0dad6de36ef3cb84c3fb87b7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a81872dc39fe443859c9a2b50091144736dee7cba1890f6cf454da842f57d930/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/96f6f112a3e5478adad6af878b9f2de63f843b7b9635ba1cac9b6e2a96d6a8d0/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/69363e6ff9623eaa4c8818dd2660c1c84e5516059068dcda9ea2022729148a39/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/502e290892b4e392459681173cc1f9aa165e4417eccf8dcd88d072351984aa48/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/82d3459de2ef7f1231f31d9bcf3e329c7eb6bd032d9c0a7dccd046e2a01b9d63/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/e67cbc480efb20644b10db96fbb38836086623b462fd290f45b8eb1c522bc2e9/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-143:{mountpoint:/var/lib/containers/storage/overlay/201ba627ddb12c096d79ced41090ba7d97240f59887b169ef18d10cf489e1a5e/merged major:0 minor:143 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/81c3972714f6debc67460419d44b8fe9ed39f66f7bcb6dbc16ca8a4aa24cf776/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/90a719c97322bef1b6d7dd0382bde18228d3d7b8baaea437f7082aaf38e1c300/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3e892fd14b0f5f52d383184d1e31241fcc36ec2492995f86fa7d68bc1cf3903a/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/aabe28746e3abbda42682ce9bcd1c28a5645942d1f53af2da8dc499114d672dd/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/ca4dfd7dce041e06368c8b40d02683f263624f46771338e03eb493bf82363a7c/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/a72fc6be21a6dd3b9cc88e70bded69ccb50bd1dfb05a1e250a4de81486396aff/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/b643bb1c4ae16bc6c0cfc794ff568c9440b1aa4d1102a8845851813f1c0b64b9/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/ef02a1e40bf0de54e0661d4db5c8e4e5915dc29c25f8dca10810357539b725c0/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/316c3678ad7b0d00818ab1f283b5983c4ed8f1ddbb0284670bf2af97a4d2a527/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-208:{mountpoint:/var/lib/containers/storage/overlay/285d05d9ad3ddf42aa3cfef7d4fbba4f31a952983b28f4d32c2c59639c864b4b/merged major:0 minor:208 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/4f060d2facf176761b5478ec10e7431a6d8b559fcf8df3fb2eb9d6b061f25dfc/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-220:{mountpoint:/var/lib/containers/storage/overlay/23d650bcc2fa75b0094282fc582e8406dc7e728f3fe27475df3f41eaabd19fcb/merged major:0 minor:220 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/4d45d32d842d5a4c67eec49dc7078b0f0d0315762cd97119f9aefd6312e38e29/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/d2ecae934192ca7a90c69ea3345fa9cc6862f48cd34700f80edf30c094371e66/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/bdfdb32ca3d89361b7e4c61a9f9b6fd7977fb6fa0a6c4f4e9e5745f7e15b13c1/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-240:{mountpoint:/var/lib/containers/storage/overlay/611e62c5706015170212fdb224bd5fc64af459c9924aa86d7bd975f8f4d1c855/merged major:0 minor:240 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/58a2717e2d5f40f23821e5a92ec76adaa8b57dcff20be360e00f1387adde17e4/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/6fa7455a88c49c3979f590b6a6e466d80c76166680389d9a24c9bbd25ba3a951/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/b51e197f59f8324bf7247be51f356037aaa65c668075e9d9d098c6cbe2cbb4d5/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-270:{mountpoint:/var/lib/containers/storage/overlay/4209ad36a1d1ed82f677631981dfcbda91a6e4db89b73163e0db7d0704d03d97/merged major:0 minor:270 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/3a51738c11ca7d9c0430237edd3d99033ed3a65af9583a2c3b6703016f442377/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/d5027dee41e03b28e30446add877d618cbc1e55c0b807a4cdd36e06e4a87f6ea/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-283:{mountpoint:/var/lib/containers/storage/overlay/19a3c4279fad5a7c763ccd798d95ac078f43aacada0a898cb9aacf9528e8fa33/merged major:0 minor:283 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/d3aa7f82c8b769b059cb6df052c79a9e035acf93b7262cfb1fd0754152e6efff/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-300:{mountpoint:/var/lib/containers/storage/overlay/724126d393000a17b19c20a8c3df9fbc0d421804a21b1059e0557ed92598607b/merged major:0 minor:300 fsType:overlay blockSize:0} overlay_0-305:{mountpoint:/var/lib/containers/storage/overlay/9db40a8df75369e763d342896b5bf43433009539c0eb211ccc739f3f5ae28ed5/merged major:0 minor:305 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/70a4a4c95d9b156408cb82a5e3d0b01c3439f6580337cb61833d10ad3f1f27d0/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-315:{mountpoint:/var/lib/containers/storage/overlay/99863487a73d058c5ee1b74a35857c29478c4136b33c74a1020f16f032573996/merged major:0 minor:315 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/09487a899924d67a65c3c5c93c73c366667b21dd2b5df7f68f25cd83e9c11ad7/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-328:{mountpoint:/var/lib/containers/storage/overlay/db779532a6a54c88e92cff5c23f8c70679459ed4d0cfed7aa19cf3c8fa07355a/merged major:0 minor:328 fsType:overlay blockSize:0} overlay_0-332:{mountpoint:/var/lib/containers/storage/overlay/c4697e122dcc5b40685f8a8ca839c2c61727986b31e46f0ec6126322b0023cce/merged major:0 minor:332 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/90021aaa5e72e7ac33f9ca1d562ab49ba21da5c2646dd32ad01add76e370bac8/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-340:{mountpoint:/var/lib/containers/storage/overlay/4033d9afafd117c71e1e575bfb55339a92c89f400c90b2ba98309519b52f43c5/merged major:0 minor:340 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/13c0c82fa051576d27b80debddf61bf727e931b585beabd8895c2d7e536c2959/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/e5e797ba7ca9dca7ff6065354281f83190d6ac796c952952188cb0d32897fa46/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/51f6ebfb7a5335a20fd4760424691240afdba99433e308b18f6f3f31b44ba779/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/ad3065bf7ebe780df7994c8461115ac2dd762a94a8ee91f47e74f4a942f09d70/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-361:{mountpoint:/var/lib/containers/storage/overlay/80168b6b0faad2380f85adaec719e1070f3a4e48220b207368acf3d3dd8bf657/merged major:0 minor:361 fsType:overlay blockSize:0} overlay_0-370:{mountpoint:/var/lib/containers/storage/overlay/4fcf44a878b5964d8763cae930e66a061791811a3592dc7fe10d6c06586116d2/merged major:0 minor:370 fsType:overlay blockSize:0} overlay_0-394:{mountpoint:/var/lib/containers/storage/overlay/ba6fdf5bf53e99259f7795c8b1df9a50e118367ff9ae0c58d24d5433bd069531/merged major:0 minor:394 fsType:overlay blockSize:0} overlay_0-396:{mountpoint:/var/lib/containers/storage/overlay/ae9c6f18d9fbdde8eb23aa12eb9a0db7a3a78144de25fdc5acddd97a73593539/merged major:0 minor:396 fsType:overlay blockSize:0} overlay_0-398:{mountpoint:/var/lib/containers/storage/overlay/491497b6f9dbb5bb99cf9cff666e21651d90f0cb47f7802c2a0662069072fd6b/merged major:0 minor:398 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/c1560a23e363357f01bd1097f7f410c5a0561aa397101ed8af4f2c9194e0a37c/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-409:{mountpoint:/var/lib/containers/storage/overlay/dd37e14bc05063af78b2f4d53055bdc4207db24a423a44e3813637cabdf6aa3a/merged major:0 minor:409 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/aa1c88cf2151484a25b6af868c85dfd549633e27533bbcc9b8184ccab469f27d/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-419:{mountpoint:/var/lib/containers/storage/overlay/f0e2a6a208a378e3c1e6a08fe0c1b0e689d3512239971f5e6a9a8df5f142579f/merged major:0 minor:419 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/c69990f71b411980d61b144bc81ec43b5e3fc1ee07f26c50cf1cb80843768fa7/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/337c74c6edabca209c5c5ccc97508b6267badf32cc985328310a70d114af8211/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/679ec46f9dd2cf3c9c03b2c8202b7a887c2dae0056a0805535af1cc3cc8536cc/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-442:{mountpoint:/var/lib/containers/storage/overlay/bf15bde76e1fc8e6accc88aebba14506527d5edbeeea08b24474d4def651b820/merged major:0 minor:442 fsType:overlay blockSize:0} overlay_0-444:{mountpoint:/var/lib/containers/storage/overlay/db29c325441f6be664b405d02f73f71a5523eee56dfafa3ae9a926bfbe612beb/merged major:0 minor:444 fsType:overlay blockSize:0} overlay_0-450:{mountpoint:/var/lib/containers/storage/overlay/aa0eaab847fff7706f1fc769fad9edb176215e3ad7db5554ac38937df74158ae/merged major:0 minor:450 fsType:overlay blockSize:0} overlay_0-455:{mountpoint:/var/lib/containers/storage/overlay/70db9c27dd3fff2b4824293c114f77f729c624b3c8d621826a3988c655a314f4/merged major:0 minor:455 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/8f7ae4886ed67ad31669e42c50633e0d372079df604d089c8e4f4811cb0b383b/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/32b9b0d89ade4831ba4af82cbee3a6e5b4b67bf1084625c8c59f0d1caeda518f/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1fc545272efb7aa18b23fc113d2778778c98be3fbf3e574b40b67e32f39fee2b/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/84b1d4c766012926ca0d0919ef204f604eac83e46c4835ebcad9877610e41e2e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/b0d95a90fd85c29dfe04f5d9f670b047e34c60baca23cb0a9e77df1623d826a4/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/e8c53266e3f57f62e394d60194f32b5ca753a6c4153a84f859dd09b40d16c966/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ad139b1ebf75e4dac2f5b9ade6960406f6354dc97d4607793a14abafd0144287/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/64e49cf4a4a1979f2ff225e9593e1123776e2c120e75b502503444f2d48470d3/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/508fa4c9ab67ed7f06cf23497fe002f9c587bffe0f4f2de4cd307fa8a435f7cf/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/7df81bfdc8a17324f525636ed2aa8ab0f3e79ba50596967610dfc7fc2593e295/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 05:31:33.498504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.498036 5565 manager.go:217] Machine: {Timestamp:2026-10-30 05:31:33.497482331 +0000 UTC m=+0.073169377 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100621 MemoryCapacity:33164333056 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26223c2377b75c779e8eff0de3832d SystemUUID:ec26223c-2377-b75c-779e-8eff0de3832d BootID:e4c37465-45b0-42bd-941f-7c0696e75f35 Filesystems:[{Device:/var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:55 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volumes/kubernetes.io~projected/kube-api-access-pw8f6 DeviceMajor:0 DeviceMinor:60 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-300 DeviceMajor:0 DeviceMinor:300 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:381 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/run/containers/storage/overlay-containers/408b223c5a0b176022fe5e0da499928d5c8a55b2fa4bfb4edfdd3e21c13c4776/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:45 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/65b4a76b7932511ece4fd3a0a2207015b2fec5d021d5877e9a0a081eb0fe8e99/userdata/shm DeviceMajor:0 DeviceMinor:212 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/config-reloader/2 DeviceMajor:0 DeviceMinor:386 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a/volumes/kubernetes.io~projected/kube-api-access-xn6rx DeviceMajor:0 DeviceMinor:66 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e/volumes/kubernetes.io~projected/kube-api-access-glnts DeviceMajor:0 DeviceMinor:67 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d6f257d286b6c3486a93c9e87262f66134e3d9f0b119afa6d605edf553162872/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:244 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:239 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582164480 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/2e463762-df03-4b5c-be5f-19892ed3d24b/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:424 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-grpc-tls DeviceMajor:0 DeviceMinor:262 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-340 DeviceMajor:0 DeviceMinor:340 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:386 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-396 DeviceMajor:0 DeviceMinor:396 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~projected/kube-api-access-nns62 DeviceMajor:0 DeviceMinor:59 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:234 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-tls DeviceMajor:0 DeviceMinor:261 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-kube-rbac-proxy DeviceMajor:0 DeviceMinor:388 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72b235d4-33f4-48d7-946b-e218318547ca/volumes/kubernetes.io~projected/kube-api-access-m2xr7 DeviceMajor:0 DeviceMinor:314 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-328 DeviceMajor:0 DeviceMinor:328 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/73be2e1a6c8ab0064fd135e40e6d045d30c76cd3483e399f09b96a33a5caa208/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-370 DeviceMajor:0 DeviceMinor:370 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/prometheus/15 DeviceMajor:0 DeviceMinor:386 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-143 DeviceMajor:0 DeviceMinor:143 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bdc5781c6ba25215772b87b1c8f2716be7030bae2f1b4f1d3b361bba24c49554/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32b15e60875a0916495e4d0c275ee79349d75af15e96bc59f0e0c30b8e608b6d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e0e3f7c9585118f18cf13b64bc5185e11437c9c717aa6773a8624040a651ccd/userdata/shm DeviceMajor:0 DeviceMinor:206 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:242 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17df51197b27dc8b85e544965173f5d9cc8a89fcb7bdfb0ed8f3349c5b1bc907/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:382 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:389 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~projected/kube-api-access-t9f99 DeviceMajor:0 DeviceMinor:233 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:201 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/config DeviceMajor:0 DeviceMinor:387 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volume-subpaths/web-config/init-config-reloader/2 DeviceMajor:0 DeviceMinor:386 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e30d6d09-f65b-4eb7-b964-7698457935cc/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-240 DeviceMajor:0 DeviceMinor:240 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d0ca187c-6933-4dd5-ada2-130897bea3ae/volumes/kubernetes.io~projected/kube-api-access-b87mf DeviceMajor:0 DeviceMinor:433 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/volumes/kubernetes.io~projected/kube-api-access-pktl4 DeviceMajor:0 DeviceMinor:203 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/54b31cad-aaef-4ade-adf9-4664d673e085/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:205 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-208 DeviceMajor:0 DeviceMinor:208 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-283 DeviceMajor:0 DeviceMinor:283 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-315 DeviceMajor:0 DeviceMinor:315 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2/userdata/shm DeviceMajor:0 DeviceMinor:392 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-419 DeviceMajor:0 DeviceMinor:419 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-thanos-sidecar-tls DeviceMajor:0 DeviceMinor:376 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-394 DeviceMajor:0 DeviceMinor:394 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/42226e87-0da6-4fa1-8d6c-842368393e3e/volumes/kubernetes.io~projected/kube-api-access-r482q DeviceMajor:0 DeviceMinor:64 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~projected/kube-api-access-g9t9s DeviceMajor:0 DeviceMinor:391 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/c6b480bd-5b3f-494c-a6e8-b59b15ba4818/volumes/kubernetes.io~projected/kube-api-access-lk7s6 DeviceMajor:0 DeviceMinor:65 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e02046b7507cfcb4af34e744c708fa7aedbb5f6a26f5b484c0948b0793bde573/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54b31cad-aaef-4ade-adf9-4664d673e085/volumes/kubernetes.io~projected/kube-api-access-2prr6 DeviceMajor:0 DeviceMinor:210 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/96dcc28eeebb0247edae000ca3859a13cadead7acf1f981ce656208ec1939a70/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:243 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-rules DeviceMajor:0 DeviceMinor:263 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:245 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:294 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bea9c6370079a3230834dfed85ed1884d5bc5defb3e5505d99b0e9b5065cb9a3/userdata/shm DeviceMajor:0 DeviceMinor:266 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:293 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/704d23ac2da779c43f7c10ad8b14d87c5afc30d867c07e3e181d2c33bde9d881/userdata/shm DeviceMajor:0 DeviceMinor:319 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-305 DeviceMajor:0 DeviceMinor:305 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-361 DeviceMajor:0 DeviceMinor:361 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/thanos-prometheus-http-client-file DeviceMajor:0 DeviceMinor:385 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:428 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/eddd45fc-666b-4ba0-a049-10623018ea60/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/4d0513be-2542-402d-bc28-d4fdea2084f1/volumes/kubernetes.io~projected/kube-api-access-9dns2 DeviceMajor:0 DeviceMinor:61 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d1738d2b2b8ccad04a58b09d6f0e400eb03b2c3e164541411448c43adb4bb03/userdata/shm DeviceMajor:0 DeviceMinor:296 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5c2bb806f0747b86bc2497e03d0b89b31df6fb59c2f9fd7dcfe5fe16f3994b65/userdata/shm DeviceMajor:0 DeviceMinor:434 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-450 DeviceMajor:0 DeviceMinor:450 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~projected/kube-api-access-mr96b DeviceMajor:0 DeviceMinor:295 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-398 DeviceMajor:0 DeviceMinor:398 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32bf9976c772697fdb86f706c8068ea680df6b04429d5509967668792b19857b/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-metrics-client-certs DeviceMajor:0 DeviceMinor:390 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582168576 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/04f43f272ee8921e1b4e62bce809a2bb412e1d1e16eefc96e2c7f8b14dc73bee/userdata/shm DeviceMajor:0 DeviceMinor:429 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f/volumes/kubernetes.io~projected/kube-api-access-4k454 DeviceMajor:0 DeviceMinor:62 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:235 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1db5f90-a5e5-40c1-b500-353c07018ff3/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:204 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/09b60a4b-f264-48b1-a862-f5a54b7316da/volumes/kubernetes.io~projected/kube-api-access-bmjc5 DeviceMajor:0 DeviceMinor:63 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d367c56784541bbf790ded0be3a761381636d46bcb4c6e84097c4d9dab849e71/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/a1db5f90-a5e5-40c1-b500-353c07018ff3/volumes/kubernetes.io~projected/kube-api-access-tj47h DeviceMajor:0 DeviceMinor:211 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:232 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy DeviceMajor:0 DeviceMinor:260 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-metrics DeviceMajor:0 DeviceMinor:264 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~projected/kube-api-access-vfwp2 DeviceMajor:0 DeviceMinor:265 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/93637adc-fd32-44b0-89ba-9a26152b5c79/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-270 DeviceMajor:0 DeviceMinor:270 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/72b235d4-33f4-48d7-946b-e218318547ca/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:311 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:246 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:383 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bc42bf-db08-410e-9eae-912afc399db4/volumes/kubernetes.io~projected/kube-api-access-t5sxt DeviceMajor:0 DeviceMinor:202 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-220 DeviceMajor:0 DeviceMinor:220 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-332 DeviceMajor:0 DeviceMinor:332 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes/kubernetes.io~secret/secret-prometheus-k8s-tls DeviceMajor:0 DeviceMinor:384 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-442 DeviceMajor:0 DeviceMinor:442 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-444 DeviceMajor:0 DeviceMinor:444 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/85bc42bf-db08-410e-9eae-912afc399db4/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/cc069a86-752c-499c-98a0-c6ad926307f7/volumes/kubernetes.io~secret/secret-thanos-querier-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:259 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:/var/lib/kubelet/pods/fdffe71b-09d7-4f44-9d2e-45cae3f24023/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:288 Capacity:31985733632 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-409 DeviceMajor:0 DeviceMinor:409 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/885cdbfc12066de80bd3661971cdcea08991931eb8ec805187dbf48fad946f6c/userdata/shm DeviceMajor:0 DeviceMinor:430 Capacity:67108864 Type:vfs Inodes:4048380 HasInodes:true} {Device:overlay_0-455 DeviceMajor:0 DeviceMinor:455 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:04f43f272ee8921 MacAddress:9a:45:4b:85:13:27 Speed:10000 Mtu:8901} {Name:32bf9976c772697 MacAddress:06:1a:59:2d:c8:eb Speed:10000 Mtu:8901} {Name:5050407a2da5116 MacAddress:36:79:4b:80:50:d3 Speed:10000 Mtu:8901} {Name:5c2bb806f0747b8 MacAddress:aa:ed:49:91:59:0b Speed:10000 Mtu:8901} {Name:65b4a76b7932511 MacAddress:36:fd:f6:3f:b2:ef Speed:10000 Mtu:8901} {Name:6d1738d2b2b8cca MacAddress:a6:cf:1b:62:a7:d2 Speed:10000 Mtu:8901} {Name:704d23ac2da779c MacAddress:b6:eb:1e:e5:b3:52 Speed:10000 Mtu:8901} {Name:885cdbfc12066de MacAddress:8e:e4:15:4f:de:c5 Speed:10000 Mtu:8901} {Name:96dcc28eeebb024 MacAddress:1e:3d:b8:c2:f7:5a Speed:10000 Mtu:8901} {Name:bc7d7503c695c86 MacAddress:e6:c6:32:56:24:fd Speed:10000 Mtu:8901} {Name:bea9c6370079a32 MacAddress:7e:8a:80:fb:af:a5 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:c9:81:61:3f Speed:0 Mtu:9001} {Name:br-int MacAddress:b2:ac:a7:a8:6a:87 Speed:0 Mtu:8901} {Name:ens5 MacAddress:02:ff:c9:81:61:3f Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:76:8d:47:63:8e:ea Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:8e:90:38:a6:6a:c0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164333056 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 05:31:33.498504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.498504 5565 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 05:31:33.498984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.498583 5565 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 05:31:33.499054 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499006 5565 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 05:31:33.499222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499057 5565 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-175.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 05:31:33.499298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499227 5565 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 05:31:33.499298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499241 5565 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 05:31:33.499298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499257 5565 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:31:33.499298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499281 5565 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 05:31:33.499466 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499451 5565 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 05:31:33.499711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499696 5565 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 05:31:33.499861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499849 5565 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 05:31:33.499918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499901 5565 kubelet.go:493] "Attempting to sync node with API server" Oct 30 05:31:33.499972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499922 5565 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 05:31:33.499972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499937 5565 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 05:31:33.499972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499952 5565 kubelet.go:405] "Adding apiserver pod source" Oct 30 05:31:33.499972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.499966 5565 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 05:31:33.500710 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.500693 5565 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:31:33.500710 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.500708 5565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:31:33.501349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.501328 5565 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 05:31:33.504037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504021 5565 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 05:31:33.504037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504035 5565 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 05:31:33.504156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504146 5565 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 05:31:33.504372 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504360 5565 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 05:31:33.504403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504390 5565 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 05:31:33.504514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504504 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504520 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504543 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504553 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504562 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504569 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504575 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504581 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504588 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504595 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504604 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 05:31:33.504610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504614 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 05:31:33.504906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504646 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 05:31:33.504906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.504655 5565 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 05:31:33.508357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.508339 5565 server.go:1263] "Started kubelet" Oct 30 05:31:33.508478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.508398 5565 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 05:31:33.508545 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.508463 5565 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 05:31:33.508600 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.508576 5565 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 05:31:33.508989 ip-10-0-129-175 systemd[1]: Started Kubernetes Kubelet. Oct 30 05:31:33.509143 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.509059 5565 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 05:31:33.509698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.509677 5565 server.go:317] "Adding debug handlers to kubelet server" Oct 30 05:31:33.517068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517019 5565 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 05:31:33.517068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517050 5565 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 05:31:33.517402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517150 5565 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 05:25:24 +0000 UTC" deadline="2028-07-04 00:48:02.231823429 +0000 UTC" Oct 30 05:31:33.517402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517183 5565 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14707h16m28.714643959s" Oct 30 05:31:33.517402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517248 5565 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 05:31:33.517402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517261 5565 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 05:31:33.517642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.517588 5565 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 05:31:33.518577 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.518519 5565 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-129-175.ec2.internal\" not found" Oct 30 05:31:33.520382 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520347 5565 factory.go:55] Registering systemd factory Oct 30 05:31:33.520382 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520380 5565 factory.go:223] Registration of the systemd container factory successfully Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520733 5565 factory.go:153] Registering CRI-O factory Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520764 5565 factory.go:223] Registration of the crio container factory successfully Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520889 5565 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520927 5565 factory.go:103] Registering Raw factory Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.520954 5565 manager.go:1199] Started watching for new ooms in manager Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521071 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-client-ca-bundle" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521122 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521140 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b60a4b-f264-48b1-a862-f5a54b7316da" volumeName="kubernetes.io/projected/09b60a4b-f264-48b1-a862-f5a54b7316da-kube-api-access-bmjc5" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521157 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521175 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521190 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42226e87-0da6-4fa1-8d6c-842368393e3e" volumeName="kubernetes.io/projected/42226e87-0da6-4fa1-8d6c-842368393e3e-kube-api-access-r482q" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521207 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1db5f90-a5e5-40c1-b500-353c07018ff3" volumeName="kubernetes.io/projected/a1db5f90-a5e5-40c1-b500-353c07018ff3-kube-api-access-tj47h" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521221 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" volumeName="kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521238 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521251 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d0513be-2542-402d-bc28-d4fdea2084f1" volumeName="kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-binary-copy" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521267 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-tls" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521282 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521294 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521307 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5234bc-f0a0-4f7f-a5b4-3443c75e274a" volumeName="kubernetes.io/projected/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-kube-api-access-xn6rx" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521326 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-rules" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521338 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddd45fc-666b-4ba0-a049-10623018ea60" volumeName="kubernetes.io/secret/eddd45fc-666b-4ba0-a049-10623018ea60-agent-certs" seLinuxMountContext="" Oct 30 05:31:33.521339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521349 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521361 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bc42bf-db08-410e-9eae-912afc399db4" volumeName="kubernetes.io/empty-dir/85bc42bf-db08-410e-9eae-912afc399db4-data-volume" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521375 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" volumeName="kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521388 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/configmap/cc069a86-752c-499c-98a0-c6ad926307f7-metrics-client-ca" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521401 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/projected/cc069a86-752c-499c-98a0-c6ad926307f7-kube-api-access-vfwp2" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521414 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/projected/fdffe71b-09d7-4f44-9d2e-45cae3f24023-kube-api-access-mr96b" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521421 5565 manager.go:319] Starting recovery of all containers Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521427 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521441 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2e463762-df03-4b5c-be5f-19892ed3d24b" volumeName="kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521485 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d0513be-2542-402d-bc28-d4fdea2084f1" volumeName="kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521498 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d0513be-2542-402d-bc28-d4fdea2084f1" volumeName="kubernetes.io/projected/4d0513be-2542-402d-bc28-d4fdea2084f1-kube-api-access-9dns2" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521511 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b31cad-aaef-4ade-adf9-4664d673e085" volumeName="kubernetes.io/secret/54b31cad-aaef-4ade-adf9-4664d673e085-metrics-tls" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521541 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72b235d4-33f4-48d7-946b-e218318547ca" volumeName="kubernetes.io/secret/72b235d4-33f4-48d7-946b-e218318547ca-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521562 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1db5f90-a5e5-40c1-b500-353c07018ff3" volumeName="kubernetes.io/secret/a1db5f90-a5e5-40c1-b500-353c07018ff3-cert" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521574 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" volumeName="kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-metrics-client-ca" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521590 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" volumeName="kubernetes.io/projected/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-kube-api-access-pktl4" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521604 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521616 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521629 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d0513be-2542-402d-bc28-d4fdea2084f1" volumeName="kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521643 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72b235d4-33f4-48d7-946b-e218318547ca" volumeName="kubernetes.io/projected/72b235d4-33f4-48d7-946b-e218318547ca-kube-api-access-m2xr7" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521656 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521672 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93637adc-fd32-44b0-89ba-9a26152b5c79" volumeName="kubernetes.io/secret/93637adc-fd32-44b0-89ba-9a26152b5c79-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521686 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" volumeName="kubernetes.io/empty-dir/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-textfile" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521707 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c6b480bd-5b3f-494c-a6e8-b59b15ba4818" volumeName="kubernetes.io/projected/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kube-api-access-lk7s6" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521721 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b60a4b-f264-48b1-a862-f5a54b7316da" volumeName="kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-cni-binary-copy" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521737 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b31cad-aaef-4ade-adf9-4664d673e085" volumeName="kubernetes.io/empty-dir/54b31cad-aaef-4ade-adf9-4664d673e085-tmp-dir" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521753 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b31cad-aaef-4ade-adf9-4664d673e085" volumeName="kubernetes.io/projected/54b31cad-aaef-4ade-adf9-4664d673e085-kube-api-access-2prr6" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521770 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" volumeName="kubernetes.io/projected/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-kube-api-access-4k454" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521788 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521803 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d0ca187c-6933-4dd5-ada2-130897bea3ae" volumeName="kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521820 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521836 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521852 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bc42bf-db08-410e-9eae-912afc399db4" volumeName="kubernetes.io/configmap/85bc42bf-db08-410e-9eae-912afc399db4-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521868 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30d6d09-f65b-4eb7-b964-7698457935cc" volumeName="kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-tuned" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521883 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="09b60a4b-f264-48b1-a862-f5a54b7316da" volumeName="kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-daemon-config" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521901 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521916 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521934 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521950 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521972 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.521986 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522013 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522029 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54b31cad-aaef-4ade-adf9-4664d673e085" volumeName="kubernetes.io/configmap/54b31cad-aaef-4ade-adf9-4664d673e085-config-volume" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522044 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522059 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93637adc-fd32-44b0-89ba-9a26152b5c79" volumeName="kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-config" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522077 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/empty-dir/cc069a86-752c-499c-98a0-c6ad926307f7-tmp" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522092 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522110 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522126 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522145 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:31:33.522124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522161 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67e26d01-cd9d-4e17-bd57-f7eb78cdc92e" volumeName="kubernetes.io/empty-dir/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-tmp-dir" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522180 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67e26d01-cd9d-4e17-bd57-f7eb78cdc92e" volumeName="kubernetes.io/projected/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-kube-api-access-glnts" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522210 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522224 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-grpc-tls" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522237 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f9e88e1-4405-45ca-8fad-9fbe5c20d41f" volumeName="kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522252 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522266 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522281 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" volumeName="kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-tls" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522295 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30d6d09-f65b-4eb7-b964-7698457935cc" volumeName="kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-tmp" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522310 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="eddd45fc-666b-4ba0-a049-10623018ea60" volumeName="kubernetes.io/configmap/eddd45fc-666b-4ba0-a049-10623018ea60-konnectivity-ca" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522323 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522338 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522350 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522365 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522378 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bc42bf-db08-410e-9eae-912afc399db4" volumeName="kubernetes.io/secret/85bc42bf-db08-410e-9eae-912afc399db4-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522390 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc069a86-752c-499c-98a0-c6ad926307f7" volumeName="kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-metrics" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522402 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522416 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522430 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="85bc42bf-db08-410e-9eae-912afc399db4" volumeName="kubernetes.io/projected/85bc42bf-db08-410e-9eae-912afc399db4-kube-api-access-t5sxt" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522443 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93637adc-fd32-44b0-89ba-9a26152b5c79" volumeName="kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-env-overrides" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522456 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522472 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522485 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93637adc-fd32-44b0-89ba-9a26152b5c79" volumeName="kubernetes.io/projected/93637adc-fd32-44b0-89ba-9a26152b5c79-kube-api-access-pw8f6" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522499 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fdffe71b-09d7-4f44-9d2e-45cae3f24023" volumeName="kubernetes.io/empty-dir/fdffe71b-09d7-4f44-9d2e-45cae3f24023-audit-log" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522516 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14cac2a8-351c-4736-8732-15cc5af686a7" volumeName="kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522549 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1c5234bc-f0a0-4f7f-a5b4-3443c75e274a" volumeName="kubernetes.io/configmap/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-iptables-alerter-script" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522563 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="42226e87-0da6-4fa1-8d6c-842368393e3e" volumeName="kubernetes.io/configmap/42226e87-0da6-4fa1-8d6c-842368393e3e-serviceca" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522578 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7473d6d4-13de-4ce5-acc3-771d24ca8b88" volumeName="kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522593 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="93637adc-fd32-44b0-89ba-9a26152b5c79" volumeName="kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-script-lib" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522608 5565 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e30d6d09-f65b-4eb7-b964-7698457935cc" volumeName="kubernetes.io/projected/e30d6d09-f65b-4eb7-b964-7698457935cc-kube-api-access-nns62" seLinuxMountContext="" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522621 5565 reconstruct.go:97] "Volume reconstruction finished" Oct 30 05:31:33.523604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.522630 5565 reconciler.go:29] "Reconciler: start to sync state" Oct 30 05:31:33.556648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.556602 5565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 05:31:33.559212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.559170 5565 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 05:31:33.559212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.559207 5565 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 05:31:33.559434 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.559232 5565 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 05:31:33.559434 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.559270 5565 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 05:31:33.570102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.570065 5565 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 05:31:33.570834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.570804 5565 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:33.571014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.570991 5565 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:33.576657 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.576623 5565 generic.go:356] "Generic (PLEG): container finished" podID="63e598fb2c0ecb0a10670947e4c1ef09" containerID="3d30e6744beac1ef0d3dd1a4ec171e397168d27324800d036cc0cae619f6a57f" exitCode=0 Oct 30 05:31:33.581736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.581705 5565 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="b8c60550f375e7ee93c448fd3b1737133fe9908d3c39c38c0a64fa9517f46992" exitCode=0 Oct 30 05:31:33.581736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.581730 5565 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="61db0e09b3acccbd1bf2a19ba1f1381519c92f2357f0b1058ec547d57a88b59e" exitCode=0 Oct 30 05:31:33.581736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.581737 5565 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="2a7f078badb7d21f0fba642f69e444f45543f9b622d681c6de48d893d522b26d" exitCode=0 Oct 30 05:31:33.581736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.581742 5565 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="1aea8620247dbba37e499e958dd802b5b26a4578f5b884461bc111ec41dc045a" exitCode=0 Oct 30 05:31:33.581736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.581748 5565 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="7f0a08e102dd06b8032200b39a622493b66f1004f68909b325118c5a1f203ab9" exitCode=0 Oct 30 05:31:33.582128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.581753 5565 generic.go:356] "Generic (PLEG): container finished" podID="4d0513be-2542-402d-bc28-d4fdea2084f1" containerID="7186d1d3b2a9561ffb28c4cf12104bda5b2524dc9f8e7a0ab0a6692a86c61771" exitCode=0 Oct 30 05:31:33.592233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.592203 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb" exitCode=0 Oct 30 05:31:33.597928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.597892 5565 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:33.598303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.598263 5565 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 05:31:33.605100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.605073 5565 generic.go:356] "Generic (PLEG): container finished" podID="bc3d7732-2c73-44cb-9ec4-f083b1b95a7a" containerID="5a127c74a18d7fff5f2a7592baf6776c7932cec3f9e2ae50eca2362c95d49678" exitCode=0 Oct 30 05:31:33.618271 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.618237 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" exitCode=0 Oct 30 05:31:33.644982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.644950 5565 manager.go:324] Recovery completed Oct 30 05:31:33.659393 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.659359 5565 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 05:31:33.668784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.668752 5565 cpu_manager.go:226] "Starting" policy="none" Oct 30 05:31:33.668784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.668771 5565 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 05:31:33.669003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.668800 5565 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 05:31:33.669003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.668962 5565 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 05:31:33.669003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.668978 5565 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 05:31:33.669003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.669001 5565 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 05:31:33.669192 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.669010 5565 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 05:31:33.669192 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.669018 5565 policy_none.go:50] "Start" Oct 30 05:31:33.669192 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.669031 5565 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 05:31:33.669192 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.669044 5565 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 05:31:33.670665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.670640 5565 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 05:31:33.670665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.670667 5565 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 05:31:33.670799 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.670677 5565 policy_none.go:44] "Start" Oct 30 05:31:33.677025 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.676983 5565 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 05:31:33.681991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.681967 5565 manager.go:341] "Starting Device Plugin manager" Oct 30 05:31:33.682506 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.682015 5565 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 05:31:33.682506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.682504 5565 server.go:87] "Starting device plugin registration server" Oct 30 05:31:33.682839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.682825 5565 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 05:31:33.682898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.682838 5565 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 05:31:33.682958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.682935 5565 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 05:31:33.683118 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.683065 5565 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 05:31:33.683233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.683130 5565 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 05:31:33.683233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.683138 5565 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 05:31:33.683341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.683199 5565 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T05:31:33.683113864Z","UUID":"b74c6f5c-0b11-4ebd-b379-3af24126210c","Handler":null,"Name":"","Endpoint":""} Oct 30 05:31:33.685109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.684892 5565 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 05:31:33.685109 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.684931 5565 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 05:31:33.685109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.684960 5565 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 05:31:33.783405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.783367 5565 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 05:31:33.785762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.785744 5565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientMemory" Oct 30 05:31:33.785861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.785779 5565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 05:31:33.785861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.785793 5565 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-129-175.ec2.internal" event="NodeHasSufficientPID" Oct 30 05:31:33.785861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.785839 5565 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.826418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.826394 5565 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.826492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.826473 5565 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.859910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.859862 5565 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal"] Oct 30 05:31:33.860545 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860494 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" event={"ID":"63e598fb2c0ecb0a10670947e4c1ef09","Type":"ContainerStarted","Data":"3acd4c554341a466a771b9535aa1dfadc0c90573342a658a789f6415964ddf8d"} Oct 30 05:31:33.860601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860554 5565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.860601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860561 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" event={"ID":"63e598fb2c0ecb0a10670947e4c1ef09","Type":"ContainerDied","Data":"3d30e6744beac1ef0d3dd1a4ec171e397168d27324800d036cc0cae619f6a57f"} Oct 30 05:31:33.860601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860581 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" event={"ID":"63e598fb2c0ecb0a10670947e4c1ef09","Type":"ContainerStarted","Data":"7e30e18b6b94aa20a667c2692c385b560b8bf65e49b19b79b53fa7bced446853"} Oct 30 05:31:33.860601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860592 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" event={"ID":"44ac8e1d4fed32053c2639c09d7471d6","Type":"ContainerStarted","Data":"19ab01aa9531f398644ed6b550cac740950a98b4b956542dbab75ee063419555"} Oct 30 05:31:33.860753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860603 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" event={"ID":"44ac8e1d4fed32053c2639c09d7471d6","Type":"ContainerStarted","Data":"c6508e3ffd4a33d1cbc57dad84276ffc581d5a9119273ba76dd459e8d4940460"} Oct 30 05:31:33.860902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.860890 5565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.886089 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.886058 5565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:33.886228 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.886125 5565 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.886690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.886679 5565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:33.886729 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:33.886705 5565 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.973099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.973065 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44ac8e1d4fed32053c2639c09d7471d6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" (UID: \"44ac8e1d4fed32053c2639c09d7471d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.973196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.973098 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:33.973196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:33.973180 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.073805 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.073709 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44ac8e1d4fed32053c2639c09d7471d6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" (UID: \"44ac8e1d4fed32053c2639c09d7471d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.073805 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.073752 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.073805 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.073786 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.074012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.073820 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/44ac8e1d4fed32053c2639c09d7471d6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" (UID: \"44ac8e1d4fed32053c2639c09d7471d6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.074012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.073846 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.074012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.073823 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63e598fb2c0ecb0a10670947e4c1ef09-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal\" (UID: \"63e598fb2c0ecb0a10670947e4c1ef09\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.500438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.500349 5565 apiserver.go:52] "Watching apiserver" Oct 30 05:31:34.560168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.560126 5565 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 05:31:34.560865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.560833 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-k22vf","openshift-multus/multus-tbp8n","openshift-ingress-canary/ingress-canary-gpq7r","openshift-cluster-node-tuning-operator/tuned-w4sqk","openshift-network-operator/iptables-alerter-dx7wt","kube-system/konnectivity-agent-lh482","openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs","openshift-multus/network-metrics-daemon-rkb2j","openshift-dns/dns-default-plfkr","openshift-dns/node-resolver-8wf8z","openshift-ovn-kubernetes/ovnkube-node-x887j","kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal","openshift-insights/insights-runtime-extractor-p69qm","openshift-monitoring/prometheus-k8s-0","kube-system/global-pull-secret-syncer-rzb85","openshift-multus/multus-additional-cni-plugins-d5q8v","openshift-monitoring/alertmanager-main-0","openshift-monitoring/thanos-querier-5b554f6f57-vkkxh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-175.ec2.internal","openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn","openshift-image-registry/node-ca-gbpks","openshift-network-diagnostics/network-check-target-lqd5k"] Oct 30 05:31:34.582037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.581997 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.582282 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582160 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.582282 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582206 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-twkbr\"" Oct 30 05:31:34.582417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582345 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pt4k5\"" Oct 30 05:31:34.582505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582488 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2nnkr\"" Oct 30 05:31:34.582584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582565 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.582584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582582 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.582683 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582006 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 05:31:34.582773 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582758 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 05:31:34.582841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582791 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.582951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.582934 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 05:31:34.583045 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583018 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-5cj6p\"" Oct 30 05:31:34.583045 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583029 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.583250 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583236 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 05:31:34.583328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583277 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-76x48\"" Oct 30 05:31:34.583376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583353 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.583486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583469 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.583600 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583585 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.583663 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583614 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.583749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583735 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 05:31:34.583822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583776 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-c6sjw\"" Oct 30 05:31:34.583994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.583978 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 05:31:34.592471 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.592434 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-88sr6\"" Oct 30 05:31:34.592471 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.592437 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-897vl\"" Oct 30 05:31:34.592741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.592640 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-w4h88\"" Oct 30 05:31:34.592741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.592659 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 05:31:34.592858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.592766 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-c4j6h\"" Oct 30 05:31:34.592925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.592851 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-wnglw\"" Oct 30 05:31:34.594060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594042 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.594159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594141 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 05:31:34.594207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594167 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 05:31:34.594285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594272 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.594335 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594307 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.594386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594349 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:31:34.594386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594354 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 05:31:34.594460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594394 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.594460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594444 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 05:31:34.594460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594448 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 05:31:34.594603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594476 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.594603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594592 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 05:31:34.594766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594612 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.594766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594614 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 05:31:34.594766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594596 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.594766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594747 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w87jr\"" Oct 30 05:31:34.594938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594776 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 05:31:34.594938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594832 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 05:31:34.595019 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.594987 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 05:31:34.598788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.598767 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.598885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.598838 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 05:31:34.598885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.598837 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.598885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.598865 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:31:34.599016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.598911 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 05:31:34.599016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.598960 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.599604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.599592 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:31:34.600139 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.600125 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 05:31:34.608989 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.608962 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 05:31:34.609161 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.609146 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 05:31:34.609254 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.609239 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 05:31:34.612061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612042 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 05:31:34.612061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612050 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:31:34.612180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612082 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 05:31:34.612180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612105 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:31:34.612180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612137 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:31:34.612180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612157 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:31:34.612735 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612714 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612756 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612780 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612796 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-lnf9b\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612792 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j88tp\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612830 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612839 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:31:34.612861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612849 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612864 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6w7wj\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612875 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612879 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612913 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612950 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612959 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612980 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.612982 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-9l0qo440ric9d\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613107 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613137 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613146 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613142 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2znb\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613177 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613189 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613236 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613242 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:31:34.613248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613244 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613268 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-9q7xl\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613276 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613280 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613302 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613316 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6avl1qpif1ntl\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613351 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613360 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6u3rkoovj0l3b\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613405 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613414 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613468 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613506 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lsrxx\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613512 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wzzmx\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613620 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-w46sp\"" Oct 30 05:31:34.613898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.613630 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-hj286\"" Oct 30 05:31:34.620630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.620607 5565 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 05:31:34.626201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.626180 5565 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.651739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.651705 5565 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 05:31:34.651891 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:34.651761 5565 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-129-175.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-175.ec2.internal" Oct 30 05:31:34.676239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676212 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.676336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676245 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.676336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676269 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-netns\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.676336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676307 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-system-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.676336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676325 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-hostroot\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.676499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676405 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.676499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676443 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1db5f90-a5e5-40c1-b500-353c07018ff3-cert\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:34.676499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676464 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-metrics-server-audit-profiles\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.676869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676483 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.676938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676897 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.676984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676942 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-ovn\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.676984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676978 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93637adc-fd32-44b0-89ba-9a26152b5c79-ovn-node-metrics-cert\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.677088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676998 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.677088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677011 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmjc5\" (UniqueName: \"kubernetes.io/projected/09b60a4b-f264-48b1-a862-f5a54b7316da-kube-api-access-bmjc5\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.677088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.676998 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.677088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677040 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.677088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677075 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.677299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677121 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.677429 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677408 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-metrics-server-audit-profiles\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.677481 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677428 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.677552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677482 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/93637adc-fd32-44b0-89ba-9a26152b5c79-ovn-node-metrics-cert\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.677609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.677549 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.680347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680320 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pktl4\" (UniqueName: \"kubernetes.io/projected/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-kube-api-access-pktl4\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.680455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680359 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-systemd\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.680455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680386 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-lib-modules\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.680455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680407 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-host\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.680625 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680456 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.680893 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680869 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85bc42bf-db08-410e-9eae-912afc399db4-data-volume\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.680953 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680925 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sxt\" (UniqueName: \"kubernetes.io/projected/85bc42bf-db08-410e-9eae-912afc399db4-kube-api-access-t5sxt\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.680953 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680929 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-accelerators-collector-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.680953 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680944 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/85bc42bf-db08-410e-9eae-912afc399db4-data-volume\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.681076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680956 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b31cad-aaef-4ade-adf9-4664d673e085-tmp-dir\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.681076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.680989 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-var-lib-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.681076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681001 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1db5f90-a5e5-40c1-b500-353c07018ff3-cert\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:34.681076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681019 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.681076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681033 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54b31cad-aaef-4ade-adf9-4664d673e085-tmp-dir\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.681076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681062 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681090 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-kubernetes\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681121 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681160 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681171 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m2xr7\" (UniqueName: \"kubernetes.io/projected/72b235d4-33f4-48d7-946b-e218318547ca-kube-api-access-m2xr7\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681218 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681239 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681281 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-iptables-alerter-script\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681293 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.681314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681306 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-textfile\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681331 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nns62\" (UniqueName: \"kubernetes.io/projected/e30d6d09-f65b-4eb7-b964-7698457935cc-kube-api-access-nns62\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681358 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-system-cni-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681384 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681409 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-multus-certs\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681428 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-iptables-alerter-script\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681436 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr96b\" (UniqueName: \"kubernetes.io/projected/fdffe71b-09d7-4f44-9d2e-45cae3f24023-kube-api-access-mr96b\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681460 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-textfile\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681522 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681617 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b31cad-aaef-4ade-adf9-4664d673e085-metrics-tls\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681637 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eddd45fc-666b-4ba0-a049-10623018ea60-konnectivity-ca\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681660 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-log-socket\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681675 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-sys\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681675 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.681711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681690 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-env-overrides\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681739 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-socket-dir-parent\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681783 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54b31cad-aaef-4ade-adf9-4664d673e085-metrics-tls\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681832 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-socket-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681851 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysconfig\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681854 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-env-overrides\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681858 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/eddd45fc-666b-4ba0-a049-10623018ea60-konnectivity-ca\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681877 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-cni-binary-copy\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.681927 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682075 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fdffe71b-09d7-4f44-9d2e-45cae3f24023-audit-log\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682077 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-cni-binary-copy\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.682160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682146 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682179 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/fdffe71b-09d7-4f44-9d2e-45cae3f24023-audit-log\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682185 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682202 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-systemd-units\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682308 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-host-slash\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682322 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682336 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682341 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682374 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-client-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682389 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682404 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glnts\" (UniqueName: \"kubernetes.io/projected/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-kube-api-access-glnts\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682418 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw8f6\" (UniqueName: \"kubernetes.io/projected/93637adc-fd32-44b0-89ba-9a26152b5c79-kube-api-access-pw8f6\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682432 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-grpc-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.682454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682450 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-etc-selinux\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682497 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682540 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682568 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682595 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwp2\" (UniqueName: \"kubernetes.io/projected/cc069a86-752c-499c-98a0-c6ad926307f7-kube-api-access-vfwp2\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682599 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682620 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-client-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682727 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682740 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-grpc-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682747 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-node-log\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682766 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-client-certs\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682789 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-daemon-config\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682817 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-wtmp\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682848 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7s6\" (UniqueName: \"kubernetes.io/projected/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kube-api-access-lk7s6\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682874 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682900 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-k8s-cni-cncf-io\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682929 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682953 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-device-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682963 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682967 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-daemon-config\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682977 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-systemd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.682905 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683014 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-metrics-client-ca\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.683081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683053 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-conf\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683107 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683152 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683192 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-metrics-client-ca\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683213 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-tls\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683219 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683291 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-tls\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683333 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683338 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683379 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683424 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-dbus\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683452 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42226e87-0da6-4fa1-8d6c-842368393e3e-host\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683501 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-tls\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683506 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683504 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r482q\" (UniqueName: \"kubernetes.io/projected/42226e87-0da6-4fa1-8d6c-842368393e3e-kube-api-access-r482q\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683619 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683659 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-modprobe-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683678 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-tuned\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683695 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683711 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85bc42bf-db08-410e-9eae-912afc399db4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683731 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-var-lib-kubelet\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683756 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683790 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-tuned\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683820 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683852 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.683911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683919 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683932 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683943 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683961 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683976 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-config\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683992 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-netns\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684007 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-multus\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684023 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.683942 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/85bc42bf-db08-410e-9eae-912afc399db4-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684061 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684111 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn6rx\" (UniqueName: \"kubernetes.io/projected/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-kube-api-access-xn6rx\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684126 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684197 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-config\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684219 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684235 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684261 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-metrics-certs\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684286 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684329 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-kubelet-config\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684337 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684351 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684373 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684400 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cc069a86-752c-499c-98a0-c6ad926307f7-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684447 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/2e463762-df03-4b5c-be5f-19892ed3d24b-original-pull-secret\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684509 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-kubelet\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684508 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684579 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2prr6\" (UniqueName: \"kubernetes.io/projected/54b31cad-aaef-4ade-adf9-4664d673e085-kube-api-access-2prr6\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684597 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684612 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k454\" (UniqueName: \"kubernetes.io/projected/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-kube-api-access-4k454\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684642 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-cnibin\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684671 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684780 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-kubelet\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684821 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85bc42bf-db08-410e-9eae-912afc399db4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684828 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/4d0513be-2542-402d-bc28-d4fdea2084f1-cni-binary-copy\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684848 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-registration-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684895 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684925 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684953 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-bin\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.684975 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-sys\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685022 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685040 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-netd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685060 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.685032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685066 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685094 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/85bc42bf-db08-410e-9eae-912afc399db4-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685117 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-root\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685145 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-tmp\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685181 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685220 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e30d6d09-f65b-4eb7-b964-7698457935cc-tmp\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685231 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685256 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-hosts-file\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685302 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eddd45fc-666b-4ba0-a049-10623018ea60-agent-certs\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685334 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-script-lib\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685341 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685360 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-cnibin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685386 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85bc42bf-db08-410e-9eae-912afc399db4-crio-socket\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685444 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc069a86-752c-499c-98a0-c6ad926307f7-metrics-client-ca\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685477 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72b235d4-33f4-48d7-946b-e218318547ca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685517 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42226e87-0da6-4fa1-8d6c-842368393e3e-serviceca\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685545 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-etc-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685555 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/93637adc-fd32-44b0-89ba-9a26152b5c79-ovnkube-script-lib\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685563 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-run\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685601 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/eddd45fc-666b-4ba0-a049-10623018ea60-agent-certs\") pod \"konnectivity-agent-lh482\" (UID: \"eddd45fc-666b-4ba0-a049-10623018ea60\") " pod="kube-system/konnectivity-agent-lh482" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685660 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dns2\" (UniqueName: \"kubernetes.io/projected/4d0513be-2542-402d-bc28-d4fdea2084f1-kube-api-access-9dns2\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685674 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cc069a86-752c-499c-98a0-c6ad926307f7-metrics-client-ca\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685692 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685721 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685754 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-bin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685771 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/72b235d4-33f4-48d7-946b-e218318547ca-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685778 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-etc-kubernetes\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685803 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-sys-fs\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685833 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685859 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685890 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54b31cad-aaef-4ade-adf9-4664d673e085-config-volume\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685926 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-slash\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685934 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fdffe71b-09d7-4f44-9d2e-45cae3f24023-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.685976 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686003 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc069a86-752c-499c-98a0-c6ad926307f7-tmp\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686024 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/42226e87-0da6-4fa1-8d6c-842368393e3e-serviceca\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686032 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686073 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686092 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-tmp-dir\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686093 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54b31cad-aaef-4ade-adf9-4664d673e085-config-volume\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686118 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686142 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686145 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-conf-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686224 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cc069a86-752c-499c-98a0-c6ad926307f7-tmp\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686236 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686255 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686289 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tj47h\" (UniqueName: \"kubernetes.io/projected/a1db5f90-a5e5-40c1-b500-353c07018ff3-kube-api-access-tj47h\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686306 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-tls\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686320 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686377 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686392 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686421 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-tmp-dir\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686634 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686668 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686703 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/fdffe71b-09d7-4f44-9d2e-45cae3f24023-secret-metrics-server-tls\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.686860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.686819 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.703872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.703846 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmjc5\" (UniqueName: \"kubernetes.io/projected/09b60a4b-f264-48b1-a862-f5a54b7316da-kube-api-access-bmjc5\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.704016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.703925 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") pod \"prometheus-k8s-0\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:34.705663 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.705640 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pktl4\" (UniqueName: \"kubernetes.io/projected/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-kube-api-access-pktl4\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.705895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.705872 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sxt\" (UniqueName: \"kubernetes.io/projected/85bc42bf-db08-410e-9eae-912afc399db4-kube-api-access-t5sxt\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.710012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.709990 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m2xr7\" (UniqueName: \"kubernetes.io/projected/72b235d4-33f4-48d7-946b-e218318547ca-kube-api-access-m2xr7\") pod \"monitoring-plugin-77f4f76594-kdkbs\" (UID: \"72b235d4-33f4-48d7-946b-e218318547ca\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:34.710493 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.710472 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr96b\" (UniqueName: \"kubernetes.io/projected/fdffe71b-09d7-4f44-9d2e-45cae3f24023-kube-api-access-mr96b\") pod \"metrics-server-6b8f6c9d6d-ttlgm\" (UID: \"fdffe71b-09d7-4f44-9d2e-45cae3f24023\") " pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:34.712140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.712116 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b87mf\" (UniqueName: \"kubernetes.io/projected/d0ca187c-6933-4dd5-ada2-130897bea3ae-kube-api-access-b87mf\") pod \"network-check-target-lqd5k\" (UID: \"d0ca187c-6933-4dd5-ada2-130897bea3ae\") " pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.712506 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r482q\" (UniqueName: \"kubernetes.io/projected/42226e87-0da6-4fa1-8d6c-842368393e3e-kube-api-access-r482q\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.712824 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7s6\" (UniqueName: \"kubernetes.io/projected/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kube-api-access-lk7s6\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.712883 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k454\" (UniqueName: \"kubernetes.io/projected/5f9e88e1-4405-45ca-8fad-9fbe5c20d41f-kube-api-access-4k454\") pod \"network-metrics-daemon-rkb2j\" (UID: \"5f9e88e1-4405-45ca-8fad-9fbe5c20d41f\") " pod="openshift-multus/network-metrics-daemon-rkb2j" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.712883 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwp2\" (UniqueName: \"kubernetes.io/projected/cc069a86-752c-499c-98a0-c6ad926307f7-kube-api-access-vfwp2\") pod \"thanos-querier-5b554f6f57-vkkxh\" (UID: \"cc069a86-752c-499c-98a0-c6ad926307f7\") " pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.713023 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tj47h\" (UniqueName: \"kubernetes.io/projected/a1db5f90-a5e5-40c1-b500-353c07018ff3-kube-api-access-tj47h\") pod \"ingress-canary-gpq7r\" (UID: \"a1db5f90-a5e5-40c1-b500-353c07018ff3\") " pod="openshift-ingress-canary/ingress-canary-gpq7r" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.713177 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw8f6\" (UniqueName: \"kubernetes.io/projected/93637adc-fd32-44b0-89ba-9a26152b5c79-kube-api-access-pw8f6\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.713357 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2prr6\" (UniqueName: \"kubernetes.io/projected/54b31cad-aaef-4ade-adf9-4664d673e085-kube-api-access-2prr6\") pod \"dns-default-plfkr\" (UID: \"54b31cad-aaef-4ade-adf9-4664d673e085\") " pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.713377 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dns2\" (UniqueName: \"kubernetes.io/projected/4d0513be-2542-402d-bc28-d4fdea2084f1-kube-api-access-9dns2\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.713436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.713394 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn6rx\" (UniqueName: \"kubernetes.io/projected/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-kube-api-access-xn6rx\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.713798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.713566 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glnts\" (UniqueName: \"kubernetes.io/projected/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-kube-api-access-glnts\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.716225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.716202 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") pod \"alertmanager-main-0\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:34.718643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.718623 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nns62\" (UniqueName: \"kubernetes.io/projected/e30d6d09-f65b-4eb7-b964-7698457935cc-kube-api-access-nns62\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.787349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787314 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-kubernetes\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787360 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-system-cni-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787383 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-multus-certs\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787407 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-log-socket\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787428 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-sys\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787436 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-system-cni-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787452 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-socket-dir-parent\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787511 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-kubernetes\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787519 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-socket-dir-parent\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787540 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-sys\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.787524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787490 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-log-socket\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787495 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-multus-certs\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787553 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-socket-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787589 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysconfig\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787605 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-socket-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787615 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-systemd-units\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787640 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-host-slash\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787663 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysconfig\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787668 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-etc-selinux\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787699 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-node-log\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787721 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-systemd-units\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787724 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-wtmp\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787755 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-node-log\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787760 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787735 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1c5234bc-f0a0-4f7f-a5b4-3443c75e274a-host-slash\") pod \"iptables-alerter-dx7wt\" (UID: \"1c5234bc-f0a0-4f7f-a5b4-3443c75e274a\") " pod="openshift-network-operator/iptables-alerter-dx7wt" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787722 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-etc-selinux\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787778 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-k8s-cni-cncf-io\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787797 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-k8s-cni-cncf-io\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787806 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787815 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-node-exporter-wtmp\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787828 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-device-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787853 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-systemd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787856 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-device-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787870 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-conf\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787895 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-dbus\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787906 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-systemd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787919 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42226e87-0da6-4fa1-8d6c-842368393e3e-host\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787941 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-modprobe-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787951 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-dbus\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-var-lib-kubelet\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787988 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/42226e87-0da6-4fa1-8d6c-842368393e3e-host\") pod \"node-ca-gbpks\" (UID: \"42226e87-0da6-4fa1-8d6c-842368393e3e\") " pod="openshift-image-registry/node-ca-gbpks" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787988 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-conf\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.788000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.787989 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788027 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-netns\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788037 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-var-lib-kubelet\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788050 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-multus\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788059 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-run-netns\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788031 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788080 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-kubelet-config\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788105 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788121 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/2e463762-df03-4b5c-be5f-19892ed3d24b-kubelet-config\") pod \"global-pull-secret-syncer-rzb85\" (UID: \"2e463762-df03-4b5c-be5f-19892ed3d24b\") " pod="kube-system/global-pull-secret-syncer-rzb85" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788129 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-kubelet\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788080 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-modprobe-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788081 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-multus\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788157 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-cnibin\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788167 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788173 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-kubelet\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788187 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/4d0513be-2542-402d-bc28-d4fdea2084f1-cnibin\") pod \"multus-additional-cni-plugins-d5q8v\" (UID: \"4d0513be-2542-402d-bc28-d4fdea2084f1\") " pod="openshift-multus/multus-additional-cni-plugins-d5q8v" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788193 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-kubelet\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788183 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-kubelet\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788217 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-registration-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788234 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788250 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-bin\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788286 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-sys\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788305 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-registration-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788315 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-netd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788358 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-netd\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788361 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-sys\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788383 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-cni-bin\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788355 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788405 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-sysctl-d\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788409 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788409 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-root\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788430 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/bc3d7732-2c73-44cb-9ec4-f083b1b95a7a-root\") pod \"node-exporter-k22vf\" (UID: \"bc3d7732-2c73-44cb-9ec4-f083b1b95a7a\") " pod="openshift-monitoring/node-exporter-k22vf" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788444 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-hosts-file\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788466 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-cnibin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788487 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85bc42bf-db08-410e-9eae-912afc399db4-crio-socket\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788514 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-etc-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788540 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-cnibin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788542 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/67e26d01-cd9d-4e17-bd57-f7eb78cdc92e-hosts-file\") pod \"node-resolver-8wf8z\" (UID: \"67e26d01-cd9d-4e17-bd57-f7eb78cdc92e\") " pod="openshift-dns/node-resolver-8wf8z" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788555 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-run\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788571 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/85bc42bf-db08-410e-9eae-912afc399db4-crio-socket\") pod \"insights-runtime-extractor-p69qm\" (UID: \"85bc42bf-db08-410e-9eae-912afc399db4\") " pod="openshift-insights/insights-runtime-extractor-p69qm" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788607 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-run\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788577 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-etc-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788581 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-bin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788610 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-host-var-lib-cni-bin\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788635 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-etc-kubernetes\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788657 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-sys-fs\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788673 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-slash\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788690 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788708 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-conf-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788725 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-sys-fs\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788734 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-netns\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788725 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-slash\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788737 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-ovn-kubernetes\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788707 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-etc-kubernetes\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788764 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-system-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788786 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-hostroot\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788766 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-host-run-netns\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788804 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-multus-conf-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788816 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-ovn\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788851 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-systemd\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788882 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-hostroot\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788914 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-lib-modules\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788934 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-run-ovn\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788980 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-etc-systemd\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788984 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-host\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.788938 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-host\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.789025 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-var-lib-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.789049 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.789077 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e30d6d09-f65b-4eb7-b964-7698457935cc-lib-modules\") pod \"tuned-w4sqk\" (UID: \"e30d6d09-f65b-4eb7-b964-7698457935cc\") " pod="openshift-cluster-node-tuning-operator/tuned-w4sqk" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.789126 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c6b480bd-5b3f-494c-a6e8-b59b15ba4818-kubelet-dir\") pod \"aws-ebs-csi-driver-node-jtkgn\" (UID: \"c6b480bd-5b3f-494c-a6e8-b59b15ba4818\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-jtkgn" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.789164 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/09b60a4b-f264-48b1-a862-f5a54b7316da-system-cni-dir\") pod \"multus-tbp8n\" (UID: \"09b60a4b-f264-48b1-a862-f5a54b7316da\") " pod="openshift-multus/multus-tbp8n" Oct 30 05:31:34.789437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.789167 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/93637adc-fd32-44b0-89ba-9a26152b5c79-var-lib-openvswitch\") pod \"ovnkube-node-x887j\" (UID: \"93637adc-fd32-44b0-89ba-9a26152b5c79\") " pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:34.882850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:34.882806 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-lh482" Oct 30 05:31:35.134027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:35.133958 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6b8f6c9d6d-ttlgm" Oct 30 05:31:36.017335 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:36.017299 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-kdkbs" Oct 30 05:31:36.528768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:36.528740 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5b554f6f57-vkkxh" Oct 30 05:31:37.692492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:37.692459 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-plfkr" Oct 30 05:31:38.048359 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:38.048333 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:42.148442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:42.148401 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:31:42.413719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:42.413644 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:44.219209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:44.219167 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:44.219726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:44.219431 5565 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 05:31:47.535303 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:47.535215 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:31:47.618582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.618546 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:47.619012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.618982 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="alertmanager" containerID="cri-o://b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1" gracePeriod=120 Oct 30 05:31:47.619083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.619015 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-metric" containerID="cri-o://7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a" gracePeriod=120 Oct 30 05:31:47.619083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.619057 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="prom-label-proxy" containerID="cri-o://d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc" gracePeriod=120 Oct 30 05:31:47.619185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.619078 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="config-reloader" containerID="cri-o://2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0" gracePeriod=120 Oct 30 05:31:47.619185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.619098 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-web" containerID="cri-o://b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc" gracePeriod=120 Oct 30 05:31:47.619185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:47.619164 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy" containerID="cri-o://20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c" gracePeriod=120 Oct 30 05:31:48.680572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680508 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc" exitCode=0 Oct 30 05:31:48.680572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680571 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a" exitCode=0 Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680569 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc"} Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680582 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c" exitCode=0 Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680669 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a"} Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680675 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0" exitCode=0 Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680688 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1" exitCode=0 Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680696 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c"} Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680721 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0"} Oct 30 05:31:48.680995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.680748 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1"} Oct 30 05:31:48.896463 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.896424 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:48.900522 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900495 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900569 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900596 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900616 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900641 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900669 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900706 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900875 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900925 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.900978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.900968 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:48.901159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.901136 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:48.901221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.901181 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:48.901280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.901254 5565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.901338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.901278 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7473d6d4-13de-4ce5-acc3-771d24ca8b88-metrics-client-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:48.903961 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.903903 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.904128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.904014 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.904205 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.904181 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.904339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.904321 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.904401 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.904311 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "kube-api-access-t9f99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:48.904401 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.904365 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:48.908211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.908181 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:48.915183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:48.915146 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:49.001636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001558 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:49.001636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001605 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:49.001855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001639 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") pod \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\" (UID: \"7473d6d4-13de-4ce5-acc3-771d24ca8b88\") " Oct 30 05:31:49.001918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001858 5565 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.001918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001880 5565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-main-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.001918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001897 5565 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-cluster-tls-config\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.001918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001912 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9f99\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-kube-api-access-t9f99\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.002069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001929 5565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.002069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001945 5565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-web-config\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.002069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001960 5565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.002069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.001974 5565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7473d6d4-13de-4ce5-acc3-771d24ca8b88-tls-assets\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.002231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.002110 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:49.003801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.003774 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:49.004115 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.004098 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric" pod "7473d6d4-13de-4ce5-acc3-771d24ca8b88" (UID: "7473d6d4-13de-4ce5-acc3-771d24ca8b88"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:49.102518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.102475 5565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-config-out\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.102518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.102511 5565 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7473d6d4-13de-4ce5-acc3-771d24ca8b88-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.102518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.102521 5565 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7473d6d4-13de-4ce5-acc3-771d24ca8b88-alertmanager-main-db\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:49.687623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.687590 5565 generic.go:356] "Generic (PLEG): container finished" podID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerID="b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc" exitCode=0 Oct 30 05:31:49.688055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.687678 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.688055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.687674 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc"} Oct 30 05:31:49.688055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.687748 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7473d6d4-13de-4ce5-acc3-771d24ca8b88","Type":"ContainerDied","Data":"5050407a2da5116d724bf211fb3c0810213f79ff7edca3f26ad7707344f564c7"} Oct 30 05:31:49.688055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.687782 5565 scope.go:122] "RemoveContainer" containerID="d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc" Oct 30 05:31:49.695726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.695707 5565 scope.go:122] "RemoveContainer" containerID="7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a" Oct 30 05:31:49.710374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.710353 5565 scope.go:122] "RemoveContainer" containerID="20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c" Oct 30 05:31:49.718174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.718150 5565 scope.go:122] "RemoveContainer" containerID="b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc" Oct 30 05:31:49.725650 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.725627 5565 scope.go:122] "RemoveContainer" containerID="2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0" Oct 30 05:31:49.733009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.732983 5565 scope.go:122] "RemoveContainer" containerID="b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1" Oct 30 05:31:49.740775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.740754 5565 scope.go:122] "RemoveContainer" containerID="b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb" Oct 30 05:31:49.748219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.748065 5565 scope.go:122] "RemoveContainer" containerID="d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc" Oct 30 05:31:49.748456 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.748430 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc\": container with ID starting with d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc not found: ID does not exist" containerID="d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc" Oct 30 05:31:49.748537 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.748466 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc"} err="failed to get container status \"d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc\": rpc error: code = NotFound desc = could not find container \"d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc\": container with ID starting with d42619360db2c70b4ca2df98eee02cc8bfda7a81e9c9ef9d689976a99c8680cc not found: ID does not exist" Oct 30 05:31:49.748589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.748541 5565 scope.go:122] "RemoveContainer" containerID="7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a" Oct 30 05:31:49.748802 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.748782 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a\": container with ID starting with 7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a not found: ID does not exist" containerID="7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a" Oct 30 05:31:49.748843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.748810 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a"} err="failed to get container status \"7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a\": rpc error: code = NotFound desc = could not find container \"7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a\": container with ID starting with 7193b511dd40a5e09d19495674173fe620ceb0600fa4aea0b22f6fc987ce8b0a not found: ID does not exist" Oct 30 05:31:49.748843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.748826 5565 scope.go:122] "RemoveContainer" containerID="20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c" Oct 30 05:31:49.749058 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.749044 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c\": container with ID starting with 20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c not found: ID does not exist" containerID="20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c" Oct 30 05:31:49.749095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749061 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c"} err="failed to get container status \"20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c\": rpc error: code = NotFound desc = could not find container \"20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c\": container with ID starting with 20bdeb4b80cb71ac5fb1e9da4ac8eb4eeb1fbd37408826f999d1ad9dd2b76c9c not found: ID does not exist" Oct 30 05:31:49.749095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749074 5565 scope.go:122] "RemoveContainer" containerID="b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc" Oct 30 05:31:49.749272 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.749257 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc\": container with ID starting with b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc not found: ID does not exist" containerID="b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc" Oct 30 05:31:49.749313 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749276 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc"} err="failed to get container status \"b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc\": rpc error: code = NotFound desc = could not find container \"b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc\": container with ID starting with b25441af8bf164180438ea5329f7b4ca35b2f32441e3caae159cc348f12ed3dc not found: ID does not exist" Oct 30 05:31:49.749313 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749287 5565 scope.go:122] "RemoveContainer" containerID="2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0" Oct 30 05:31:49.749489 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.749474 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0\": container with ID starting with 2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0 not found: ID does not exist" containerID="2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0" Oct 30 05:31:49.749540 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749493 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0"} err="failed to get container status \"2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0\": rpc error: code = NotFound desc = could not find container \"2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0\": container with ID starting with 2551c4b94db390961dac42f9d8b5e5fa1a035499e79626047eba81d37ca635c0 not found: ID does not exist" Oct 30 05:31:49.749540 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749504 5565 scope.go:122] "RemoveContainer" containerID="b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1" Oct 30 05:31:49.749704 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.749690 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1\": container with ID starting with b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1 not found: ID does not exist" containerID="b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1" Oct 30 05:31:49.749743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749706 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1"} err="failed to get container status \"b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1\": rpc error: code = NotFound desc = could not find container \"b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1\": container with ID starting with b62310e9f83b83a2b29b8656a3a9cbda8148e6c97033252210bd3f75bd3a82f1 not found: ID does not exist" Oct 30 05:31:49.749743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749717 5565 scope.go:122] "RemoveContainer" containerID="b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb" Oct 30 05:31:49.749951 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:49.749935 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb\": container with ID starting with b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb not found: ID does not exist" containerID="b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb" Oct 30 05:31:49.749999 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.749953 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb"} err="failed to get container status \"b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb\": rpc error: code = NotFound desc = could not find container \"b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb\": container with ID starting with b506d778a9b25e777808cddf2ac3830a008c4341ed6c8b38ba506556a7b82cdb not found: ID does not exist" Oct 30 05:31:49.765167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.765138 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:49.779119 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.779093 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:49.811935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.811903 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:49.812174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812162 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="config-reloader" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812175 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="config-reloader" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812187 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812192 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812200 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="alertmanager" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812205 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="alertmanager" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812215 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-metric" Oct 30 05:31:49.812221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812220 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-metric" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812231 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-web" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812236 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-web" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812249 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="init-config-reloader" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812254 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="init-config-reloader" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812262 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="prom-label-proxy" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812266 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="prom-label-proxy" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812312 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="config-reloader" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812319 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812325 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="alertmanager" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812331 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-metric" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812336 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="kube-rbac-proxy-web" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812343 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="init-config-reloader" Oct 30 05:31:49.812452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.812348 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" containerName="prom-label-proxy" Oct 30 05:31:49.815749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.815727 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.848775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848721 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 05:31:49.848915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848824 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 05:31:49.848915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848859 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 05:31:49.848915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848905 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 05:31:49.849073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848925 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 05:31:49.849073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848941 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 05:31:49.849073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848943 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-lsrxx\"" Oct 30 05:31:49.849073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848964 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 05:31:49.849073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.848969 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 05:31:49.849073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.849003 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 05:31:49.851237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.851218 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.908890 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0073ab4b-bf58-4c99-81d6-049a441f9d96-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.908952 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0073ab4b-bf58-4c99-81d6-049a441f9d96-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909004 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909035 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909076 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-config-volume\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909101 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909132 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rscjg\" (UniqueName: \"kubernetes.io/projected/0073ab4b-bf58-4c99-81d6-049a441f9d96-kube-api-access-rscjg\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909179 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909211 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0073ab4b-bf58-4c99-81d6-049a441f9d96-config-out\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909241 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909274 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0073ab4b-bf58-4c99-81d6-049a441f9d96-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909312 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0073ab4b-bf58-4c99-81d6-049a441f9d96-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:49.911272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:49.909347 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-web-config\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010319 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-config-volume\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010373 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010401 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rscjg\" (UniqueName: \"kubernetes.io/projected/0073ab4b-bf58-4c99-81d6-049a441f9d96-kube-api-access-rscjg\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010435 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010458 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0073ab4b-bf58-4c99-81d6-049a441f9d96-config-out\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010479 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010509 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0073ab4b-bf58-4c99-81d6-049a441f9d96-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010562 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0073ab4b-bf58-4c99-81d6-049a441f9d96-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010591 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-web-config\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010626 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0073ab4b-bf58-4c99-81d6-049a441f9d96-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010674 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0073ab4b-bf58-4c99-81d6-049a441f9d96-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010721 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010744 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.010806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.010795 5565 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 05:31:50.011424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.011388 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0073ab4b-bf58-4c99-81d6-049a441f9d96-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.011834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.011646 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0073ab4b-bf58-4c99-81d6-049a441f9d96-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.011834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.011669 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/0073ab4b-bf58-4c99-81d6-049a441f9d96-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.013863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.013689 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-config-volume\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.013863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.013729 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.014033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.013867 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/0073ab4b-bf58-4c99-81d6-049a441f9d96-tls-assets\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.014033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.013870 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.014033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.013972 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-web-config\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.014187 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.014138 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/0073ab4b-bf58-4c99-81d6-049a441f9d96-config-out\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.014328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.014304 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.014384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.014370 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.015556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.015519 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0073ab4b-bf58-4c99-81d6-049a441f9d96-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.037140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.037111 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rscjg\" (UniqueName: \"kubernetes.io/projected/0073ab4b-bf58-4c99-81d6-049a441f9d96-kube-api-access-rscjg\") pod \"alertmanager-main-0\" (UID: \"0073ab4b-bf58-4c99-81d6-049a441f9d96\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.132126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.132033 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 05:31:50.468033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.467953 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 05:31:50.471849 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:50.471811 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0073ab4b_bf58_4c99_81d6_049a441f9d96.slice/crio-c2c686e268aaaee41820bead76996889a8b4c78ce41ffba6f160d3e58f406bad WatchSource:0}: Error finding container c2c686e268aaaee41820bead76996889a8b4c78ce41ffba6f160d3e58f406bad: Status 404 returned error can't find the container with id c2c686e268aaaee41820bead76996889a8b4c78ce41ffba6f160d3e58f406bad Oct 30 05:31:50.693648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.693612 5565 generic.go:356] "Generic (PLEG): container finished" podID="0073ab4b-bf58-4c99-81d6-049a441f9d96" containerID="008176f249ea584acbe7b5957794a5d3628f071d35be5797c8491de7368c2789" exitCode=0 Oct 30 05:31:50.694124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.693648 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerDied","Data":"008176f249ea584acbe7b5957794a5d3628f071d35be5797c8491de7368c2789"} Oct 30 05:31:50.694124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:50.693707 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"c2c686e268aaaee41820bead76996889a8b4c78ce41ffba6f160d3e58f406bad"} Oct 30 05:31:51.512963 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.512927 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-56556f6c49-cqssd"] Oct 30 05:31:51.515242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.515224 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520471 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520444 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-secret-telemeter-client\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520498 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-federate-client-tls\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520577 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-metrics-client-ca\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520627 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-serving-certs-ca-bundle\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520739 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-telemeter-client-tls\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520771 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv5wg\" (UniqueName: \"kubernetes.io/projected/8fa63c73-1e28-4ef7-9672-29a5a36997d9-kube-api-access-bv5wg\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520803 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.520913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.520835 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.539207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.539184 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 05:31:51.539343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.539326 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 05:31:51.539438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.539426 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 05:31:51.539480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.539430 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 05:31:51.539601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.539586 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-gr7tm\"" Oct 30 05:31:51.558495 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.558472 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56556f6c49-cqssd"] Oct 30 05:31:51.564754 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.564732 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7473d6d4-13de-4ce5-acc3-771d24ca8b88" path="/var/lib/kubelet/pods/7473d6d4-13de-4ce5-acc3-771d24ca8b88/volumes" Oct 30 05:31:51.585619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.585595 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 05:31:51.602760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.602737 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 05:31:51.621604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621575 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-serving-certs-ca-bundle\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621629 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-telemeter-client-tls\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621658 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv5wg\" (UniqueName: \"kubernetes.io/projected/8fa63c73-1e28-4ef7-9672-29a5a36997d9-kube-api-access-bv5wg\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621686 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621715 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621758 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-secret-telemeter-client\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621807 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-federate-client-tls\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.621872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.621831 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-metrics-client-ca\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.622403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.622378 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-serving-certs-ca-bundle\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.622567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.622513 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-metrics-client-ca\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.622840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.622815 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8fa63c73-1e28-4ef7-9672-29a5a36997d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.624239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.624214 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.624324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.624244 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-federate-client-tls\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.624362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.624322 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-secret-telemeter-client\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.624419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.624404 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/8fa63c73-1e28-4ef7-9672-29a5a36997d9-telemeter-client-tls\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.653596 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.653573 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv5wg\" (UniqueName: \"kubernetes.io/projected/8fa63c73-1e28-4ef7-9672-29a5a36997d9-kube-api-access-bv5wg\") pod \"telemeter-client-56556f6c49-cqssd\" (UID: \"8fa63c73-1e28-4ef7-9672-29a5a36997d9\") " pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:51.701440 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.701411 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"334e0f8422f37d5575a40ded7f56760bb0da7657a2bbe519f0528d0be73589bd"} Oct 30 05:31:51.701761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.701495 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"a76077aa901fd7bb001056ec92bd078f7fc3834295e18443dc9bd2ab3306ff80"} Oct 30 05:31:51.701761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.701521 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"025cfefb4a5245c3d169af0d359797c8c99d32d5633949ef446ab1968a088698"} Oct 30 05:31:51.701761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.701560 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"f7549da65000770a1154b1a5c00ab87f7a46b6285834d64db68ae797e885960c"} Oct 30 05:31:51.701761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.701583 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"bd4b296c2b6a2aad9c1be0470da33346517687614582acd0f4a4697235227130"} Oct 30 05:31:51.701761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.701605 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"0073ab4b-bf58-4c99-81d6-049a441f9d96","Type":"ContainerStarted","Data":"584952477b4242deec173592f5a07f41f82869962eb9e37cb7457091af5f3ca1"} Oct 30 05:31:51.761066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.761016 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.747944334 podStartE2EDuration="2.761004478s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.060144ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:51.760865477 +0000 UTC m=+18.336552532" watchObservedRunningTime="2026-10-30 05:31:51.761004478 +0000 UTC m=+18.336691511" Oct 30 05:31:51.827695 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:51.827622 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" Oct 30 05:31:52.155022 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.154936 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-56556f6c49-cqssd"] Oct 30 05:31:52.158404 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:52.158379 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fa63c73_1e28_4ef7_9672_29a5a36997d9.slice/crio-1135dcc427640ec6796cdf0c7cfacc3e8eaf4453d3f57d4a35c164c71c8f7ecb WatchSource:0}: Error finding container 1135dcc427640ec6796cdf0c7cfacc3e8eaf4453d3f57d4a35c164c71c8f7ecb: Status 404 returned error can't find the container with id 1135dcc427640ec6796cdf0c7cfacc3e8eaf4453d3f57d4a35c164c71c8f7ecb Oct 30 05:31:52.160437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.160418 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:31:52.216723 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.216692 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:52.217024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.216999 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="prometheus" containerID="cri-o://9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" gracePeriod=600 Oct 30 05:31:52.217120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.217028 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="thanos-sidecar" containerID="cri-o://fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" gracePeriod=600 Oct 30 05:31:52.217120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.217027 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy" containerID="cri-o://e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" gracePeriod=600 Oct 30 05:31:52.217120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.217066 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-web" containerID="cri-o://c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" gracePeriod=600 Oct 30 05:31:52.217120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.217084 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="config-reloader" containerID="cri-o://b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" gracePeriod=600 Oct 30 05:31:52.217307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.217091 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" gracePeriod=600 Oct 30 05:31:52.445751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.445725 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.528557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528515 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.528803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528598 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.528803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528633 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.528803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528663 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.528803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528696 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.528803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528737 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.528803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528772 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.529107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528805 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.529107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528846 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.529107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.528891 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.529107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.529047 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.529300 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.529211 5565 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.529408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.529353 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.529560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.529488 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.529986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.529961 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:52.533339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.533280 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.533449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.533363 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.533516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.533446 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.533576 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.533506 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.534297 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.533780 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "kube-api-access-g9t9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:52.534618 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.534598 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:31:52.629513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629478 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629547 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629578 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629603 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629638 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629655 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629672 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629696 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629724 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.629755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629746 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") pod \"14cac2a8-351c-4736-8732-15cc5af686a7\" (UID: \"14cac2a8-351c-4736-8732-15cc5af686a7\") " Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.629997 5565 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-tls-assets\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630035 5565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630054 5565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630069 5565 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630086 5565 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630103 5565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630117 5565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-db\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630132 5565 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630146 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9t9s\" (UniqueName: \"kubernetes.io/projected/14cac2a8-351c-4736-8732-15cc5af686a7-kube-api-access-g9t9s\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.630332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630097 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.630936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630502 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.630936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.630694 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.631723 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.631674 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:31:52.632671 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.632647 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.632780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.632715 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:31:52.632845 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.632787 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.634211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.634192 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.634378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.634355 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.642842 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.642822 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config" pod "14cac2a8-351c-4736-8732-15cc5af686a7" (UID: "14cac2a8-351c-4736-8732-15cc5af686a7"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:31:52.710896 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710864 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" exitCode=0 Oct 30 05:31:52.710896 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710895 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" exitCode=0 Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710905 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" exitCode=0 Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710915 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" exitCode=0 Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710927 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" exitCode=0 Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710935 5565 generic.go:356] "Generic (PLEG): container finished" podID="14cac2a8-351c-4736-8732-15cc5af686a7" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" exitCode=0 Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.710958 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711042 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711114 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711156 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711195 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711235 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} Oct 30 05:31:52.711295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711274 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} Oct 30 05:31:52.711701 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711314 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"14cac2a8-351c-4736-8732-15cc5af686a7","Type":"ContainerDied","Data":"bc7d7503c695c86d8f209e091fa6eb02649fd4cd21dcabdeca04a183589e66c2"} Oct 30 05:31:52.711701 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.711359 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.713448 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.713413 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" event={"ID":"8fa63c73-1e28-4ef7-9672-29a5a36997d9","Type":"ContainerStarted","Data":"1135dcc427640ec6796cdf0c7cfacc3e8eaf4453d3f57d4a35c164c71c8f7ecb"} Oct 30 05:31:52.723217 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.723199 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.731027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.730985 5565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731010 5565 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-kube-rbac-proxy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731020 5565 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-config\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731030 5565 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731039 5565 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-metrics-client-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731048 5565 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-grpc-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731058 5565 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/14cac2a8-351c-4736-8732-15cc5af686a7-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731067 5565 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-secret-metrics-client-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731075 5565 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/14cac2a8-351c-4736-8732-15cc5af686a7-config-out\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731084 5565 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/14cac2a8-351c-4736-8732-15cc5af686a7-web-config\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:31:52.731249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.731240 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.737971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.737954 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.744666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.744649 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.751076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.751060 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.759157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.759077 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.766604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.766588 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.766847 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.766831 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.766890 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.766854 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} err="failed to get container status \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" Oct 30 05:31:52.766890 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.766870 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.767109 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.767091 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.767157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.767112 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} err="failed to get container status \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" Oct 30 05:31:52.767157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.767125 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.767756 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.767344 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.767756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.767366 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} err="failed to get container status \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" Oct 30 05:31:52.767756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.767386 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.767756 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.767683 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.767756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.767709 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} err="failed to get container status \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" Oct 30 05:31:52.767756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.767730 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.768044 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.768023 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.768098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768051 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} err="failed to get container status \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" Oct 30 05:31:52.768098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768085 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.768365 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.768343 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.768432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768374 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} err="failed to get container status \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" Oct 30 05:31:52.768432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768393 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.768694 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:31:52.768677 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.768764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768702 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} err="failed to get container status \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" Oct 30 05:31:52.768764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768721 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.768993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768973 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} err="failed to get container status \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" Oct 30 05:31:52.768993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.768992 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.769278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.769256 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} err="failed to get container status \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" Oct 30 05:31:52.769338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.769281 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.769520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.769499 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} err="failed to get container status \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" Oct 30 05:31:52.769520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.769521 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.769846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.769825 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} err="failed to get container status \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" Oct 30 05:31:52.769922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.769846 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.770109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.770091 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} err="failed to get container status \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" Oct 30 05:31:52.770186 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.770110 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.770428 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.770408 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} err="failed to get container status \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" Oct 30 05:31:52.770493 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.770428 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.770731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.770711 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} err="failed to get container status \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" Oct 30 05:31:52.770817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.770733 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.771019 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771001 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} err="failed to get container status \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" Oct 30 05:31:52.771082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771020 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.771287 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771250 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} err="failed to get container status \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" Oct 30 05:31:52.771287 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771276 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.771564 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771520 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} err="failed to get container status \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" Oct 30 05:31:52.771663 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771562 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.771874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771831 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} err="failed to get container status \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" Oct 30 05:31:52.771874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.771859 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.772140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.772113 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} err="failed to get container status \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" Oct 30 05:31:52.772140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.772137 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.772465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.772435 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} err="failed to get container status \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" Oct 30 05:31:52.772465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.772464 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.772813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.772723 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} err="failed to get container status \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" Oct 30 05:31:52.772813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.772746 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.773078 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773050 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} err="failed to get container status \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" Oct 30 05:31:52.773078 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773074 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.773370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773349 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} err="failed to get container status \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" Oct 30 05:31:52.773438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773371 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.773556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773537 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:52.773669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773638 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} err="failed to get container status \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" Oct 30 05:31:52.773669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773663 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.773966 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773942 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} err="failed to get container status \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" Oct 30 05:31:52.774045 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.773966 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.774324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.774285 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} err="failed to get container status \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" Oct 30 05:31:52.774324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.774308 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.774603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.774581 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} err="failed to get container status \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" Oct 30 05:31:52.774603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.774603 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.774843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.774820 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} err="failed to get container status \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" Oct 30 05:31:52.774916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.774876 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.775137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775114 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} err="failed to get container status \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" Oct 30 05:31:52.775211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775138 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.775387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775357 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} err="failed to get container status \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" Oct 30 05:31:52.775387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775383 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.775678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775644 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} err="failed to get container status \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" Oct 30 05:31:52.775678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775671 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.775933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775913 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} err="failed to get container status \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" Oct 30 05:31:52.775998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.775934 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.776167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.776150 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} err="failed to get container status \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" Oct 30 05:31:52.776225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.776168 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.776442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.776418 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} err="failed to get container status \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" Oct 30 05:31:52.776442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.776441 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.776974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.776741 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} err="failed to get container status \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" Oct 30 05:31:52.776974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.776765 5565 scope.go:122] "RemoveContainer" containerID="ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509" Oct 30 05:31:52.777225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.777203 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509"} err="failed to get container status \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": rpc error: code = NotFound desc = could not find container \"ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509\": container with ID starting with ca8b8151acadd423d07fdb1700933dd5a51a51c74884e8767d427e4d715c9509 not found: ID does not exist" Oct 30 05:31:52.777225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.777224 5565 scope.go:122] "RemoveContainer" containerID="e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9" Oct 30 05:31:52.777485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.777463 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9"} err="failed to get container status \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": rpc error: code = NotFound desc = could not find container \"e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9\": container with ID starting with e9e0bccabfa471d1b5cba6b0e4d551614fc58507f1cb31333cbed0a31381b7e9 not found: ID does not exist" Oct 30 05:31:52.777584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.777485 5565 scope.go:122] "RemoveContainer" containerID="c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8" Oct 30 05:31:52.777728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.777706 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8"} err="failed to get container status \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": rpc error: code = NotFound desc = could not find container \"c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8\": container with ID starting with c610d136779e0c2586232b5e07fa6b1522a7eaf0d8bb9764a3f2ff3e403abfc8 not found: ID does not exist" Oct 30 05:31:52.777806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.777728 5565 scope.go:122] "RemoveContainer" containerID="fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac" Oct 30 05:31:52.778058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778041 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:52.778130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778049 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac"} err="failed to get container status \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": rpc error: code = NotFound desc = could not find container \"fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac\": container with ID starting with fec7e4e4368d9316574954e591882f7c219aa49ded38c5fbdf05113824d299ac not found: ID does not exist" Oct 30 05:31:52.778130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778072 5565 scope.go:122] "RemoveContainer" containerID="b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee" Oct 30 05:31:52.778400 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778382 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee"} err="failed to get container status \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": rpc error: code = NotFound desc = could not find container \"b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee\": container with ID starting with b155d8f03781b2a95348a31e772061c92ffdbf16b0b6ebb47be8cce7fb2cb0ee not found: ID does not exist" Oct 30 05:31:52.778400 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778399 5565 scope.go:122] "RemoveContainer" containerID="9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0" Oct 30 05:31:52.778675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778651 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0"} err="failed to get container status \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": rpc error: code = NotFound desc = could not find container \"9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0\": container with ID starting with 9f6144c5f03489d358b2ac520d196cafa3b0482b910c72b1188607119a5d07a0 not found: ID does not exist" Oct 30 05:31:52.778675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778675 5565 scope.go:122] "RemoveContainer" containerID="278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839" Oct 30 05:31:52.778924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.778898 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839"} err="failed to get container status \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": rpc error: code = NotFound desc = could not find container \"278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839\": container with ID starting with 278c5a5bcc575988e3006781d0d9f9c496aae7b9c27a95bf9677c83372509839 not found: ID does not exist" Oct 30 05:31:52.810494 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810470 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:52.810891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810855 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810892 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810902 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="prometheus" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810907 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="prometheus" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810916 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="init-config-reloader" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810921 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="init-config-reloader" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810940 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-thanos" Oct 30 05:31:52.810959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810948 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-thanos" Oct 30 05:31:52.811190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810969 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="config-reloader" Oct 30 05:31:52.811190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810983 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="config-reloader" Oct 30 05:31:52.811190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.810995 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-web" Oct 30 05:31:52.811190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.811000 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-web" Oct 30 05:31:52.811190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.811007 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="thanos-sidecar" Oct 30 05:31:52.811190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.811011 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="thanos-sidecar" Oct 30 05:31:52.812218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812192 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-web" Oct 30 05:31:52.812218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812220 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="thanos-sidecar" Oct 30 05:31:52.812395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812236 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy" Oct 30 05:31:52.812395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812255 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="prometheus" Oct 30 05:31:52.812395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812270 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="init-config-reloader" Oct 30 05:31:52.812395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812279 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="kube-rbac-proxy-thanos" Oct 30 05:31:52.812395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.812292 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" containerName="config-reloader" Oct 30 05:31:52.819590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.819570 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.831955 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.831929 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832059 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.831999 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832108 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832084 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832153 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832123 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832182 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d44bc086-596f-42d2-bfa2-ad0fae67e64a-config-out\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832246 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832202 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832273 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832318 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832427 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832384 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-web-config\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832427 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832405 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d44bc086-596f-42d2-bfa2-ad0fae67e64a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832437 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-config\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832471 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdx45\" (UniqueName: \"kubernetes.io/projected/d44bc086-596f-42d2-bfa2-ad0fae67e64a-kube-api-access-qdx45\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832575 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832610 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832720 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832695 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832732 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832759 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832782 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832808 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.832906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.832835 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.839698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839679 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Oct 30 05:31:52.839872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839756 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Oct 30 05:31:52.839872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839763 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Oct 30 05:31:52.839872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839830 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Oct 30 05:31:52.839872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839763 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Oct 30 05:31:52.839872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839767 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-6w7wj\"" Oct 30 05:31:52.839872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839852 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839880 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839897 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839908 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.839916 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.840083 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.840087 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-6avl1qpif1ntl\"" Oct 30 05:31:52.840374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.840250 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Oct 30 05:31:52.854788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.854762 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Oct 30 05:31:52.854788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.854781 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Oct 30 05:31:52.855947 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.855931 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:52.933768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.933731 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.933946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.933777 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.933946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.933808 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.933946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.933837 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.933946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.933863 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d44bc086-596f-42d2-bfa2-ad0fae67e64a-config-out\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.933980 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934013 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934042 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934094 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-web-config\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934114 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d44bc086-596f-42d2-bfa2-ad0fae67e64a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934138 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-config\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934160 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdx45\" (UniqueName: \"kubernetes.io/projected/d44bc086-596f-42d2-bfa2-ad0fae67e64a-kube-api-access-qdx45\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934196 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934244 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934277 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934312 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934342 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934364 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934391 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934422 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934469 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.934860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.934704 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.935582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.935555 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.935708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.935669 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.936690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.936621 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.937291 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.937270 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.937579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.937558 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.939642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.939579 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.940135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.940108 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d44bc086-596f-42d2-bfa2-ad0fae67e64a-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.940621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.940596 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.940827 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/d44bc086-596f-42d2-bfa2-ad0fae67e64a-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.941049 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.941265 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-config\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.941343 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.941344 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.941634 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.941867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.941825 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.944078 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.944058 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d44bc086-596f-42d2-bfa2-ad0fae67e64a-config-out\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.947132 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.947111 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d44bc086-596f-42d2-bfa2-ad0fae67e64a-web-config\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:52.962377 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:52.962328 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdx45\" (UniqueName: \"kubernetes.io/projected/d44bc086-596f-42d2-bfa2-ad0fae67e64a-kube-api-access-qdx45\") pod \"prometheus-k8s-0\" (UID: \"d44bc086-596f-42d2-bfa2-ad0fae67e64a\") " pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:53.147389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:53.147354 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:31:53.491237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:53.491204 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Oct 30 05:31:53.494396 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:31:53.494366 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44bc086_596f_42d2_bfa2_ad0fae67e64a.slice/crio-249c761447c3bee6ff978fbb980ba80010e960e08921d0dd80d3a326becfc918 WatchSource:0}: Error finding container 249c761447c3bee6ff978fbb980ba80010e960e08921d0dd80d3a326becfc918: Status 404 returned error can't find the container with id 249c761447c3bee6ff978fbb980ba80010e960e08921d0dd80d3a326becfc918 Oct 30 05:31:53.565597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:53.565567 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14cac2a8-351c-4736-8732-15cc5af686a7" path="/var/lib/kubelet/pods/14cac2a8-351c-4736-8732-15cc5af686a7/volumes" Oct 30 05:31:53.718673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:53.718645 5565 generic.go:356] "Generic (PLEG): container finished" podID="d44bc086-596f-42d2-bfa2-ad0fae67e64a" containerID="c6dc5295036ec0f46dcc97ff5be2a1d634d39516422a9e720a786de2d462b982" exitCode=0 Oct 30 05:31:53.718969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:53.718695 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerDied","Data":"c6dc5295036ec0f46dcc97ff5be2a1d634d39516422a9e720a786de2d462b982"} Oct 30 05:31:53.718969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:53.718743 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"249c761447c3bee6ff978fbb980ba80010e960e08921d0dd80d3a326becfc918"} Oct 30 05:31:54.281720 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.281687 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-x887j" Oct 30 05:31:54.724561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.724458 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" event={"ID":"8fa63c73-1e28-4ef7-9672-29a5a36997d9","Type":"ContainerStarted","Data":"848f15b829fdd97646dd027cfd2cc87ec8a2918f78e15f7b0a8bd9d0185d1031"} Oct 30 05:31:54.724561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.724523 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" event={"ID":"8fa63c73-1e28-4ef7-9672-29a5a36997d9","Type":"ContainerStarted","Data":"1c9dacd9540d7cede469455010c55c9a02c26ea2da91d15819fd76c7dcafeb82"} Oct 30 05:31:54.725030 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.724566 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" event={"ID":"8fa63c73-1e28-4ef7-9672-29a5a36997d9","Type":"ContainerStarted","Data":"a55c8c2743d80f7e8a86a307af2d9e5d510271de34535bc612cfcc95f35e2b63"} Oct 30 05:31:54.727413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.727389 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"9b84210800acec67c885932a54bc9b8c69107c74471ed394627b64f7484955e2"} Oct 30 05:31:54.727550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.727453 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"c7c506799578ee62e1424e39a8dfed456c332839e1c3b1794f4fca898d41d216"} Oct 30 05:31:54.727550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.727494 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"efe13f02b032aa02f8e252a8eb5c8060b4eb09b48eb8c2f8725a0044e6e8c6ef"} Oct 30 05:31:54.727656 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.727551 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"2279b2ddf4e230694760e83b4d5ec70c2df4990f0102355b1718503dde9f29a9"} Oct 30 05:31:54.727656 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.727590 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"9640cba811fd2062bcdc7ae006a48d218f1555cf4a0cd6411793844ec0f8f6fe"} Oct 30 05:31:54.727656 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.727634 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"d44bc086-596f-42d2-bfa2-ad0fae67e64a","Type":"ContainerStarted","Data":"ed132d4b05289fad2b56e5fffb5ef3510bd2bd3954e1c27b8c6419f50da49009"} Oct 30 05:31:54.782145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.782082 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-56556f6c49-cqssd" podStartSLOduration=2.271726738 podStartE2EDuration="3.782067786s" totalImagesPullingTime="1.510341048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:54.78171383 +0000 UTC m=+21.357400883" watchObservedRunningTime="2026-10-30 05:31:54.782067786 +0000 UTC m=+21.357754833" Oct 30 05:31:54.844884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:54.844817 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.828186329 podStartE2EDuration="2.844800911s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.614582ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:31:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:31:54.843519404 +0000 UTC m=+21.419206480" watchObservedRunningTime="2026-10-30 05:31:54.844800911 +0000 UTC m=+21.420487964" Oct 30 05:31:58.148146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:31:58.148110 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:02.151430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:32:02.151394 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lqd5k" Oct 30 05:32:17.530372 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:32:17.530326 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:32:32.969727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:32:32.969695 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:32.984976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:32:32.984954 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:33.162810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:32:33.162784 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Oct 30 05:32:47.559131 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:32:47.559093 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:33:17.529576 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:33:17.529469 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:33:47.529632 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:33:47.529598 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:34:17.530665 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:34:17.530630 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:34:47.530831 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:34:47.530755 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:35:17.529884 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:35:17.529847 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:35:47.529522 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:35:47.529484 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:36:15.865684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.865591 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-spx7h"] Oct 30 05:36:15.868952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.868922 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:15.900943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.900908 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 05:36:15.901137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.900950 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 05:36:15.901137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.900955 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-vj7xk\"" Oct 30 05:36:15.901892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.901873 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-spx7h"] Oct 30 05:36:15.934877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.934825 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hbqw\" (UniqueName: \"kubernetes.io/projected/888adb96-2bac-4bac-a858-0f5553a041f3-kube-api-access-9hbqw\") pod \"openshift-lws-operator-8465994787-spx7h\" (UID: \"888adb96-2bac-4bac-a858-0f5553a041f3\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:15.935087 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:15.934900 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888adb96-2bac-4bac-a858-0f5553a041f3-tmp\") pod \"openshift-lws-operator-8465994787-spx7h\" (UID: \"888adb96-2bac-4bac-a858-0f5553a041f3\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:16.035903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.035859 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hbqw\" (UniqueName: \"kubernetes.io/projected/888adb96-2bac-4bac-a858-0f5553a041f3-kube-api-access-9hbqw\") pod \"openshift-lws-operator-8465994787-spx7h\" (UID: \"888adb96-2bac-4bac-a858-0f5553a041f3\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:16.036100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.035923 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888adb96-2bac-4bac-a858-0f5553a041f3-tmp\") pod \"openshift-lws-operator-8465994787-spx7h\" (UID: \"888adb96-2bac-4bac-a858-0f5553a041f3\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:16.036320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.036301 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/888adb96-2bac-4bac-a858-0f5553a041f3-tmp\") pod \"openshift-lws-operator-8465994787-spx7h\" (UID: \"888adb96-2bac-4bac-a858-0f5553a041f3\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:16.061742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.061703 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hbqw\" (UniqueName: \"kubernetes.io/projected/888adb96-2bac-4bac-a858-0f5553a041f3-kube-api-access-9hbqw\") pod \"openshift-lws-operator-8465994787-spx7h\" (UID: \"888adb96-2bac-4bac-a858-0f5553a041f3\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:16.198116 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.198021 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" Oct 30 05:36:16.503794 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.503766 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-spx7h"] Oct 30 05:36:16.507201 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:36:16.507166 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod888adb96_2bac_4bac_a858_0f5553a041f3.slice/crio-5e67d792cf0f9dcc7fd5be2744ba8d71b43543de626101ca727868993c29524a WatchSource:0}: Error finding container 5e67d792cf0f9dcc7fd5be2744ba8d71b43543de626101ca727868993c29524a: Status 404 returned error can't find the container with id 5e67d792cf0f9dcc7fd5be2744ba8d71b43543de626101ca727868993c29524a Oct 30 05:36:16.604109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:16.604071 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" event={"ID":"888adb96-2bac-4bac-a858-0f5553a041f3","Type":"ContainerStarted","Data":"5e67d792cf0f9dcc7fd5be2744ba8d71b43543de626101ca727868993c29524a"} Oct 30 05:36:17.541147 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:36:17.541112 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:36:19.616883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:19.616844 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" event={"ID":"888adb96-2bac-4bac-a858-0f5553a041f3","Type":"ContainerStarted","Data":"a73e6a29050d37db0e296eb28f1747de990bed9dfa2bbfe51dd74a24a74d0a87"} Oct 30 05:36:19.664250 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:19.664194 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-spx7h" podStartSLOduration=2.023098921 podStartE2EDuration="4.664182941s" totalImagesPullingTime="2.64108402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:19.663671872 +0000 UTC m=+286.239358926" watchObservedRunningTime="2026-10-30 05:36:19.664182941 +0000 UTC m=+286.239869995" Oct 30 05:36:32.922128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.922097 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx"] Oct 30 05:36:32.925880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.925863 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:32.962462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.962435 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 05:36:32.962462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.962437 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 05:36:32.962462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.962440 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-zvlk9\"" Oct 30 05:36:32.965875 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.965855 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 05:36:32.969746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.969722 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx"] Oct 30 05:36:32.979326 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.979303 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-cert\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:32.979422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.979354 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-metrics-cert\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:32.979464 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.979452 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-manager-config\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:32.979519 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:32.979501 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swmc2\" (UniqueName: \"kubernetes.io/projected/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-kube-api-access-swmc2\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.080856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.080819 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swmc2\" (UniqueName: \"kubernetes.io/projected/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-kube-api-access-swmc2\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.081044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.080872 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-cert\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.081044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.080910 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-metrics-cert\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.081044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.080942 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-manager-config\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.081745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.081721 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-manager-config\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.083432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.083405 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-cert\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.083546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.083459 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-metrics-cert\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.106591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.106551 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-swmc2\" (UniqueName: \"kubernetes.io/projected/4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac-kube-api-access-swmc2\") pod \"lws-controller-manager-58867fb74d-n5glx\" (UID: \"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac\") " pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.237213 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.237120 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:33.536597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.536572 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx"] Oct 30 05:36:33.539098 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:36:33.539068 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd74ad6_7ad7_4b5f_9a5a_cd496acdeeac.slice/crio-3d3d65797fc05e17dff1fdafefd8a90744ad6bcc17d721248c57d6f7f2d25a11 WatchSource:0}: Error finding container 3d3d65797fc05e17dff1fdafefd8a90744ad6bcc17d721248c57d6f7f2d25a11: Status 404 returned error can't find the container with id 3d3d65797fc05e17dff1fdafefd8a90744ad6bcc17d721248c57d6f7f2d25a11 Oct 30 05:36:33.664715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:33.664676 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" event={"ID":"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac","Type":"ContainerStarted","Data":"3d3d65797fc05e17dff1fdafefd8a90744ad6bcc17d721248c57d6f7f2d25a11"} Oct 30 05:36:34.099814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.099782 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh"] Oct 30 05:36:34.104463 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.104441 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.171882 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.171848 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh"] Oct 30 05:36:34.191077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.191041 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmh5\" (UniqueName: \"kubernetes.io/projected/b79bb003-300f-46d9-b5e3-3a3289d1aee9-kube-api-access-zqmh5\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.191242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.191113 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.191242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.191204 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b79bb003-300f-46d9-b5e3-3a3289d1aee9-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.191341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.191251 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.191394 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.191345 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b79bb003-300f-46d9-b5e3-3a3289d1aee9-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.191394 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.191376 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.211768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.211731 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 05:36:34.211768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.211733 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 05:36:34.211768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.211744 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 05:36:34.212027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.211797 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 05:36:34.212027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.211847 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 05:36:34.212027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.211904 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wrhw7\"" Oct 30 05:36:34.292708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.292670 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmh5\" (UniqueName: \"kubernetes.io/projected/b79bb003-300f-46d9-b5e3-3a3289d1aee9-kube-api-access-zqmh5\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.292886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.292723 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.292886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.292755 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b79bb003-300f-46d9-b5e3-3a3289d1aee9-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.292886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.292790 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.293033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.292909 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b79bb003-300f-46d9-b5e3-3a3289d1aee9-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.293033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.292941 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.293450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.293306 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b79bb003-300f-46d9-b5e3-3a3289d1aee9-manifests\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.295154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.295127 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b79bb003-300f-46d9-b5e3-3a3289d1aee9-tmp\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.295556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.295516 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.295848 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.295827 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-webhook-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.295909 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.295841 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b79bb003-300f-46d9-b5e3-3a3289d1aee9-metrics-cert\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.317432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.317397 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmh5\" (UniqueName: \"kubernetes.io/projected/b79bb003-300f-46d9-b5e3-3a3289d1aee9-kube-api-access-zqmh5\") pod \"opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh\" (UID: \"b79bb003-300f-46d9-b5e3-3a3289d1aee9\") " pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.420503 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.420423 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:34.782836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:34.782805 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh"] Oct 30 05:36:34.786929 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:36:34.786882 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb79bb003_300f_46d9_b5e3_3a3289d1aee9.slice/crio-cdf6b6b0ed7ade4e9423aa9969e4f4ab10447d9817fe6b5d3335031a46b2e1b2 WatchSource:0}: Error finding container cdf6b6b0ed7ade4e9423aa9969e4f4ab10447d9817fe6b5d3335031a46b2e1b2: Status 404 returned error can't find the container with id cdf6b6b0ed7ade4e9423aa9969e4f4ab10447d9817fe6b5d3335031a46b2e1b2 Oct 30 05:36:35.675167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:35.675081 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" event={"ID":"4bd74ad6-7ad7-4b5f-9a5a-cd496acdeeac","Type":"ContainerStarted","Data":"88adc10d64228e5e51b4d0d4ef48b520bba240a5fab6c993af37787ccdfc20ce"} Oct 30 05:36:35.675680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:35.675243 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:35.676811 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:35.676753 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" event={"ID":"b79bb003-300f-46d9-b5e3-3a3289d1aee9","Type":"ContainerStarted","Data":"cdf6b6b0ed7ade4e9423aa9969e4f4ab10447d9817fe6b5d3335031a46b2e1b2"} Oct 30 05:36:35.740591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:35.740510 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" podStartSLOduration=1.964475256 podStartE2EDuration="3.740487006s" totalImagesPullingTime="1.77601175s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:35.738731399 +0000 UTC m=+302.314418460" watchObservedRunningTime="2026-10-30 05:36:35.740487006 +0000 UTC m=+302.316174061" Oct 30 05:36:37.685723 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:37.685684 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" event={"ID":"b79bb003-300f-46d9-b5e3-3a3289d1aee9","Type":"ContainerStarted","Data":"b291ca0e99f4671f608460536a6f42b7598b8ce7d9da8a555ded1a009260f028"} Oct 30 05:36:38.691487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:38.691457 5565 generic.go:356] "Generic (PLEG): container finished" podID="b79bb003-300f-46d9-b5e3-3a3289d1aee9" containerID="b291ca0e99f4671f608460536a6f42b7598b8ce7d9da8a555ded1a009260f028" exitCode=0 Oct 30 05:36:38.691988 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:38.691564 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" event={"ID":"b79bb003-300f-46d9-b5e3-3a3289d1aee9","Type":"ContainerDied","Data":"b291ca0e99f4671f608460536a6f42b7598b8ce7d9da8a555ded1a009260f028"} Oct 30 05:36:39.697259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:39.697221 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" event={"ID":"b79bb003-300f-46d9-b5e3-3a3289d1aee9","Type":"ContainerStarted","Data":"c43c71fd14d35f1e66ceeed613fe4e9f8b48c05795ee2a9a110e988c1bd09d35"} Oct 30 05:36:39.697712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:39.697381 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:39.800518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:39.800465 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" podStartSLOduration=2.19800404 podStartE2EDuration="5.800451822s" totalImagesPullingTime="3.155347903s" totalInitContainerRuntime="447.099879ms" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:39.78570077 +0000 UTC m=+306.361387824" watchObservedRunningTime="2026-10-30 05:36:39.800451822 +0000 UTC m=+306.376138876" Oct 30 05:36:46.683049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:46.683015 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-58867fb74d-n5glx" Oct 30 05:36:47.532035 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:36:47.531998 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:36:50.702701 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:50.702673 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh" Oct 30 05:36:55.878360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.878330 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-788bd9946-f8sxq"] Oct 30 05:36:55.885026 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.885006 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:55.902891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.902865 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788bd9946-f8sxq"] Oct 30 05:36:55.915880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.915857 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 05:36:55.916016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.915938 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 05:36:55.977200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.977172 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-tls-certs\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:55.977319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.977221 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54jvt\" (UniqueName: \"kubernetes.io/projected/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-kube-api-access-54jvt\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:55.977319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.977243 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-tmp\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:55.981979 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.981959 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 05:36:55.982041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.981978 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 05:36:55.982041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:55.981963 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-22vcj\"" Oct 30 05:36:56.078671 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.078640 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-tls-certs\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.078786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.078697 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54jvt\" (UniqueName: \"kubernetes.io/projected/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-kube-api-access-54jvt\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.078786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.078716 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-tmp\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.080874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.080852 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-tmp\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.081081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.081065 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-tls-certs\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.103907 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.103883 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54jvt\" (UniqueName: \"kubernetes.io/projected/80ec3dc9-bb84-47af-89d2-6db6ffe9df05-kube-api-access-54jvt\") pod \"kube-auth-proxy-788bd9946-f8sxq\" (UID: \"80ec3dc9-bb84-47af-89d2-6db6ffe9df05\") " pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.196081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.196027 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" Oct 30 05:36:56.493266 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.493048 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-788bd9946-f8sxq"] Oct 30 05:36:56.495233 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:36:56.495199 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ec3dc9_bb84_47af_89d2_6db6ffe9df05.slice/crio-37634f437b2f285b0480e173471bdd465a3c7490fc98fe924ce9024986db1141 WatchSource:0}: Error finding container 37634f437b2f285b0480e173471bdd465a3c7490fc98fe924ce9024986db1141: Status 404 returned error can't find the container with id 37634f437b2f285b0480e173471bdd465a3c7490fc98fe924ce9024986db1141 Oct 30 05:36:56.497212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.497198 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:36:56.760752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:56.760722 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" event={"ID":"80ec3dc9-bb84-47af-89d2-6db6ffe9df05","Type":"ContainerStarted","Data":"37634f437b2f285b0480e173471bdd465a3c7490fc98fe924ce9024986db1141"} Oct 30 05:36:59.775376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:59.775329 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" event={"ID":"80ec3dc9-bb84-47af-89d2-6db6ffe9df05","Type":"ContainerStarted","Data":"ebf487b1e082d3a77f8b3013b51a2d0b621004a74f14db9584378038ac4b53d4"} Oct 30 05:36:59.820906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:36:59.820863 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-788bd9946-f8sxq" podStartSLOduration=2.461836263 podStartE2EDuration="4.820850791s" totalImagesPullingTime="2.359014528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:36:59.820411859 +0000 UTC m=+326.396098912" watchObservedRunningTime="2026-10-30 05:36:59.820850791 +0000 UTC m=+326.396537844" Oct 30 05:37:17.558801 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:37:17.558760 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:37:47.530539 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:37:47.530444 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:38:17.530583 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:38:17.530548 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:38:47.530585 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:38:47.530523 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:39:10.978691 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:10.978654 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-pwttx"] Oct 30 05:39:10.982219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:10.982195 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:11.002033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.002013 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-x65m9\"" Oct 30 05:39:11.016008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.015985 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 05:39:11.016008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.015985 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 05:39:11.017142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.017121 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-pwttx"] Oct 30 05:39:11.037818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.037795 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh5cm\" (UniqueName: \"kubernetes.io/projected/db0127c0-1218-470b-9769-4ae07d0b80ff-kube-api-access-hh5cm\") pod \"authorino-operator-68d6dd76dc-pwttx\" (UID: \"db0127c0-1218-470b-9769-4ae07d0b80ff\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:11.139024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.138999 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh5cm\" (UniqueName: \"kubernetes.io/projected/db0127c0-1218-470b-9769-4ae07d0b80ff-kube-api-access-hh5cm\") pod \"authorino-operator-68d6dd76dc-pwttx\" (UID: \"db0127c0-1218-470b-9769-4ae07d0b80ff\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:11.168570 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.168519 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh5cm\" (UniqueName: \"kubernetes.io/projected/db0127c0-1218-470b-9769-4ae07d0b80ff-kube-api-access-hh5cm\") pod \"authorino-operator-68d6dd76dc-pwttx\" (UID: \"db0127c0-1218-470b-9769-4ae07d0b80ff\") " pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:11.294806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.294786 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:11.595415 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:11.595346 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-68d6dd76dc-pwttx"] Oct 30 05:39:11.598299 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:39:11.598272 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0127c0_1218_470b_9769_4ae07d0b80ff.slice/crio-e9acf1fa726d42e381008fefdda30d0bc2479ac85d2e122282ec9dd2194c6067 WatchSource:0}: Error finding container e9acf1fa726d42e381008fefdda30d0bc2479ac85d2e122282ec9dd2194c6067: Status 404 returned error can't find the container with id e9acf1fa726d42e381008fefdda30d0bc2479ac85d2e122282ec9dd2194c6067 Oct 30 05:39:12.287038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:12.286998 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" event={"ID":"db0127c0-1218-470b-9769-4ae07d0b80ff","Type":"ContainerStarted","Data":"e9acf1fa726d42e381008fefdda30d0bc2479ac85d2e122282ec9dd2194c6067"} Oct 30 05:39:14.297248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:14.297216 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" event={"ID":"db0127c0-1218-470b-9769-4ae07d0b80ff","Type":"ContainerStarted","Data":"a135598a5e6ea1cb127f9baa20f0bcee0f85e0078dfa19774d530ed3ca3be583"} Oct 30 05:39:14.297637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:14.297348 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:14.351841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:14.351795 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" podStartSLOduration=2.398255951 podStartE2EDuration="4.351784564s" totalImagesPullingTime="1.953528613s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:39:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:39:14.35156687 +0000 UTC m=+460.927253924" watchObservedRunningTime="2026-10-30 05:39:14.351784564 +0000 UTC m=+460.927471619" Oct 30 05:39:17.530847 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:39:17.530810 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:39:25.305197 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:39:25.305167 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-68d6dd76dc-pwttx" Oct 30 05:39:47.530366 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:39:47.530320 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:40:17.531276 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:40:17.531227 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:40:17.966463 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:17.966381 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:40:17.969722 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:17.969699 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.005184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.005154 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 30 05:40:18.005353 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.005157 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pljfh\"" Oct 30 05:40:18.006150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.006126 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:40:18.063027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.062996 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:40:18.123523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.123484 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f-config-file\") pod \"limitador-limitador-7c9b87d574-ct2m7\" (UID: \"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.123709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.123524 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n64qf\" (UniqueName: \"kubernetes.io/projected/7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f-kube-api-access-n64qf\") pod \"limitador-limitador-7c9b87d574-ct2m7\" (UID: \"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.224082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.223982 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f-config-file\") pod \"limitador-limitador-7c9b87d574-ct2m7\" (UID: \"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.224082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.224027 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n64qf\" (UniqueName: \"kubernetes.io/projected/7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f-kube-api-access-n64qf\") pod \"limitador-limitador-7c9b87d574-ct2m7\" (UID: \"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.224711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.224691 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f-config-file\") pod \"limitador-limitador-7c9b87d574-ct2m7\" (UID: \"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.249905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.249873 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n64qf\" (UniqueName: \"kubernetes.io/projected/7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f-kube-api-access-n64qf\") pod \"limitador-limitador-7c9b87d574-ct2m7\" (UID: \"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.281433 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.281395 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:18.581783 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.581760 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:40:18.583668 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:40:18.583638 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7c62a87f_5f27_403b_8cf9_e2a71b2dfd6f.slice/crio-ef3b9f6361797ee7180a4885c6ab03b56dacb88730b077ef2a3b9976c9b1b030 WatchSource:0}: Error finding container ef3b9f6361797ee7180a4885c6ab03b56dacb88730b077ef2a3b9976c9b1b030: Status 404 returned error can't find the container with id ef3b9f6361797ee7180a4885c6ab03b56dacb88730b077ef2a3b9976c9b1b030 Oct 30 05:40:18.766226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.766198 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nwsgq"] Oct 30 05:40:18.770651 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.770635 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:18.787151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.787129 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nwsgq"] Oct 30 05:40:18.787855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.787839 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pr45g\"" Oct 30 05:40:18.928155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:18.928077 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95fw\" (UniqueName: \"kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw\") pod \"authorino-56c6c9ddd5-nwsgq\" (UID: \"53321b0b-05e8-4b8e-a3ba-613536fc469c\") " pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:19.029310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.029276 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95fw\" (UniqueName: \"kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw\") pod \"authorino-56c6c9ddd5-nwsgq\" (UID: \"53321b0b-05e8-4b8e-a3ba-613536fc469c\") " pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:19.055024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.054991 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95fw\" (UniqueName: \"kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw\") pod \"authorino-56c6c9ddd5-nwsgq\" (UID: \"53321b0b-05e8-4b8e-a3ba-613536fc469c\") " pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:19.082647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.082623 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:19.380278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.380256 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nwsgq"] Oct 30 05:40:19.383451 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:40:19.383423 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod53321b0b_05e8_4b8e_a3ba_613536fc469c.slice/crio-69089de88e5e645a0109b1fbcc5cf879ab77894593bad881dabe704e362dff5d WatchSource:0}: Error finding container 69089de88e5e645a0109b1fbcc5cf879ab77894593bad881dabe704e362dff5d: Status 404 returned error can't find the container with id 69089de88e5e645a0109b1fbcc5cf879ab77894593bad881dabe704e362dff5d Oct 30 05:40:19.564290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.564225 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:19.564422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.564285 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" event={"ID":"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f","Type":"ContainerStarted","Data":"60897f2fe79d5e29384444182c900b92c6a711b0715976e6d70b0dfbb09c1b1c"} Oct 30 05:40:19.564422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.564313 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" event={"ID":"7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f","Type":"ContainerStarted","Data":"ef3b9f6361797ee7180a4885c6ab03b56dacb88730b077ef2a3b9976c9b1b030"} Oct 30 05:40:19.564422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.564332 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" event={"ID":"53321b0b-05e8-4b8e-a3ba-613536fc469c","Type":"ContainerStarted","Data":"69089de88e5e645a0109b1fbcc5cf879ab77894593bad881dabe704e362dff5d"} Oct 30 05:40:19.607954 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:19.607911 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" podStartSLOduration=1.8793755650000001 podStartE2EDuration="2.60789838s" totalImagesPullingTime="728.522815ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:19.607089744 +0000 UTC m=+526.182776800" watchObservedRunningTime="2026-10-30 05:40:19.60789838 +0000 UTC m=+526.183585434" Oct 30 05:40:21.572464 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:21.572431 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" event={"ID":"53321b0b-05e8-4b8e-a3ba-613536fc469c","Type":"ContainerStarted","Data":"c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d"} Oct 30 05:40:21.615374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:21.615331 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" podStartSLOduration=2.275793206 podStartE2EDuration="3.61531884s" totalImagesPullingTime="1.339525634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:40:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:40:21.6149326 +0000 UTC m=+528.190619657" watchObservedRunningTime="2026-10-30 05:40:21.61531884 +0000 UTC m=+528.191005897" Oct 30 05:40:30.568744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:30.568710 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-ct2m7" Oct 30 05:40:47.531128 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:40:47.531021 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:40:49.716255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.716213 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hbjsz"] Oct 30 05:40:49.719774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.719752 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:49.738344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.738322 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 05:40:49.752395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.752368 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hbjsz"] Oct 30 05:40:49.769238 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.769217 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwmlp\" (UniqueName: \"kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp\") pod \"authorino-6889cbd8c4-hbjsz\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:49.769342 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.769247 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert\") pod \"authorino-6889cbd8c4-hbjsz\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:49.806391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.806366 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hbjsz"] Oct 30 05:40:49.806557 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:40:49.806539 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-bwmlp tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" podUID="89a36694-055c-41a1-8da6-3492b4ebade1" Oct 30 05:40:49.870288 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.870264 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwmlp\" (UniqueName: \"kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp\") pod \"authorino-6889cbd8c4-hbjsz\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:49.870400 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.870301 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert\") pod \"authorino-6889cbd8c4-hbjsz\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:49.872793 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.872770 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert\") pod \"authorino-6889cbd8c4-hbjsz\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:49.899033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:49.899012 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwmlp\" (UniqueName: \"kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp\") pod \"authorino-6889cbd8c4-hbjsz\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:50.689973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.689944 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:50.697050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.697028 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:50.776564 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.776542 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert\" (UniqueName: \"kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert\") pod \"89a36694-055c-41a1-8da6-3492b4ebade1\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " Oct 30 05:40:50.776916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.776576 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp\" (UniqueName: \"kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp\") pod \"89a36694-055c-41a1-8da6-3492b4ebade1\" (UID: \"89a36694-055c-41a1-8da6-3492b4ebade1\") " Oct 30 05:40:50.778524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.778501 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp" pod "89a36694-055c-41a1-8da6-3492b4ebade1" (UID: "89a36694-055c-41a1-8da6-3492b4ebade1"). InnerVolumeSpecName "kube-api-access-bwmlp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:50.778649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.778628 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert" pod "89a36694-055c-41a1-8da6-3492b4ebade1" (UID: "89a36694-055c-41a1-8da6-3492b4ebade1"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:40:50.877479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.877456 5565 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/89a36694-055c-41a1-8da6-3492b4ebade1-tls-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:40:50.877479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:50.877478 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwmlp\" (UniqueName: \"kubernetes.io/projected/89a36694-055c-41a1-8da6-3492b4ebade1-kube-api-access-bwmlp\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:40:51.693513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:51.693484 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-hbjsz" Oct 30 05:40:51.737274 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:51.737247 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nwsgq"] Oct 30 05:40:51.737432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:51.737415 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" podUID="53321b0b-05e8-4b8e-a3ba-613536fc469c" containerName="authorino" containerID="cri-o://c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d" gracePeriod=30 Oct 30 05:40:51.779062 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:51.779036 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hbjsz"] Oct 30 05:40:51.786125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:51.786102 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-hbjsz"] Oct 30 05:40:51.977349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:51.977328 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:52.087414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.087387 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw\" (UniqueName: \"kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw\") pod \"53321b0b-05e8-4b8e-a3ba-613536fc469c\" (UID: \"53321b0b-05e8-4b8e-a3ba-613536fc469c\") " Oct 30 05:40:52.089573 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.089497 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw" pod "53321b0b-05e8-4b8e-a3ba-613536fc469c" (UID: "53321b0b-05e8-4b8e-a3ba-613536fc469c"). InnerVolumeSpecName "kube-api-access-z95fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:40:52.188707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.188650 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95fw\" (UniqueName: \"kubernetes.io/projected/53321b0b-05e8-4b8e-a3ba-613536fc469c-kube-api-access-z95fw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:40:52.698770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.698738 5565 generic.go:356] "Generic (PLEG): container finished" podID="53321b0b-05e8-4b8e-a3ba-613536fc469c" containerID="c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d" exitCode=0 Oct 30 05:40:52.698962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.698779 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" Oct 30 05:40:52.698962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.698815 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" event={"ID":"53321b0b-05e8-4b8e-a3ba-613536fc469c","Type":"ContainerDied","Data":"c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d"} Oct 30 05:40:52.698962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.698856 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-nwsgq" event={"ID":"53321b0b-05e8-4b8e-a3ba-613536fc469c","Type":"ContainerDied","Data":"69089de88e5e645a0109b1fbcc5cf879ab77894593bad881dabe704e362dff5d"} Oct 30 05:40:52.698962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.698873 5565 scope.go:122] "RemoveContainer" containerID="c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d" Oct 30 05:40:52.706903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.706887 5565 scope.go:122] "RemoveContainer" containerID="c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d" Oct 30 05:40:52.707170 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:40:52.707152 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d\": container with ID starting with c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d not found: ID does not exist" containerID="c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d" Oct 30 05:40:52.707210 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.707178 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d"} err="failed to get container status \"c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d\": rpc error: code = NotFound desc = could not find container \"c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d\": container with ID starting with c38d6c965a9ffcd962725c7adbc5ea536c211e5b1657dc3e690aa296e139045d not found: ID does not exist" Oct 30 05:40:52.764377 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.764345 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nwsgq"] Oct 30 05:40:52.772401 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:52.772374 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-nwsgq"] Oct 30 05:40:53.565043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:53.565013 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53321b0b-05e8-4b8e-a3ba-613536fc469c" path="/var/lib/kubelet/pods/53321b0b-05e8-4b8e-a3ba-613536fc469c/volumes" Oct 30 05:40:53.565391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:40:53.565350 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89a36694-055c-41a1-8da6-3492b4ebade1" path="/var/lib/kubelet/pods/89a36694-055c-41a1-8da6-3492b4ebade1/volumes" Oct 30 05:41:17.531918 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:41:17.531874 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:41:35.455360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:41:35.455316 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:41:47.531551 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:41:47.531482 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:42:17.531439 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:42:17.531399 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:42:35.989927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:35.989886 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:42:38.908990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.908958 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl"] Oct 30 05:42:38.909480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.909320 5565 cpu_manager.go:414] "containerMap: removing container" podUID="53321b0b-05e8-4b8e-a3ba-613536fc469c" containerName="authorino" Oct 30 05:42:38.909480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.909334 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53321b0b-05e8-4b8e-a3ba-613536fc469c" containerName="authorino" Oct 30 05:42:38.909480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.909396 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="53321b0b-05e8-4b8e-a3ba-613536fc469c" containerName="authorino" Oct 30 05:42:38.912472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.912452 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:38.943344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.943319 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bgqg5\"" Oct 30 05:42:38.943487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.943396 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 05:42:38.943487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.943416 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 30 05:42:38.943487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.943485 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 05:42:38.944805 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.944785 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl"] Oct 30 05:42:38.963599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.963579 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:38.963702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.963606 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:38.963702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.963643 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:38.963816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.963767 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:38.963866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.963819 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvpxk\" (UniqueName: \"kubernetes.io/projected/e413ad81-bb52-45f7-86c4-7127a1564739-kube-api-access-dvpxk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:38.963923 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:38.963901 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e413ad81-bb52-45f7-86c4-7127a1564739-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064220 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064197 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e413ad81-bb52-45f7-86c4-7127a1564739-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064326 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064242 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064326 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064262 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064326 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064297 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064326 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064312 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064335 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvpxk\" (UniqueName: \"kubernetes.io/projected/e413ad81-bb52-45f7-86c4-7127a1564739-kube-api-access-dvpxk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064682 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064808 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064744 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.064808 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.064793 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.066637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.066615 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e413ad81-bb52-45f7-86c4-7127a1564739-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.066797 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.066780 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e413ad81-bb52-45f7-86c4-7127a1564739-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.089076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.089050 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvpxk\" (UniqueName: \"kubernetes.io/projected/e413ad81-bb52-45f7-86c4-7127a1564739-kube-api-access-dvpxk\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-9hmzl\" (UID: \"e413ad81-bb52-45f7-86c4-7127a1564739\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.225960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.225895 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:39.522787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.522762 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl"] Oct 30 05:42:39.525155 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:42:39.525126 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode413ad81_bb52_45f7_86c4_7127a1564739.slice/crio-5eec7cc42c8827179109ee0aa84a4b3842bdc7bf383e4f8a358bf3c03022cab6 WatchSource:0}: Error finding container 5eec7cc42c8827179109ee0aa84a4b3842bdc7bf383e4f8a358bf3c03022cab6: Status 404 returned error can't find the container with id 5eec7cc42c8827179109ee0aa84a4b3842bdc7bf383e4f8a358bf3c03022cab6 Oct 30 05:42:39.526488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.526470 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:42:39.784945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:39.784863 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:42:40.151632 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:40.151552 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" event={"ID":"e413ad81-bb52-45f7-86c4-7127a1564739","Type":"ContainerStarted","Data":"5eec7cc42c8827179109ee0aa84a4b3842bdc7bf383e4f8a358bf3c03022cab6"} Oct 30 05:42:41.158212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.158173 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" event={"ID":"e413ad81-bb52-45f7-86c4-7127a1564739","Type":"ContainerStarted","Data":"13399bd89e582d40e05b58ab2fbce4b0f56bff0a21d9be8f5fd675c14e0703a8"} Oct 30 05:42:41.204877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.204830 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" podStartSLOduration=2.151110749 podStartE2EDuration="3.204817322s" totalImagesPullingTime="1.053706573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:41.203896711 +0000 UTC m=+667.779583766" watchObservedRunningTime="2026-10-30 05:42:41.204817322 +0000 UTC m=+667.780504372" Oct 30 05:42:41.226395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.226367 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:41.861442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.861411 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5"] Oct 30 05:42:41.864635 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.864615 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.881338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.881314 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 30 05:42:41.887776 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.887755 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.887838 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.887794 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.887907 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.887889 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.887948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.887921 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930df857-6e36-4c04-a098-ccec27be1dec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.887998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.887986 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.888051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.888040 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-298px\" (UniqueName: \"kubernetes.io/projected/930df857-6e36-4c04-a098-ccec27be1dec-kube-api-access-298px\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.895977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.895954 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5"] Oct 30 05:42:41.989278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989244 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989289 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-298px\" (UniqueName: \"kubernetes.io/projected/930df857-6e36-4c04-a098-ccec27be1dec-kube-api-access-298px\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989327 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989344 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989369 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989398 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930df857-6e36-4c04-a098-ccec27be1dec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989713 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989740 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.989780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.989771 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.991779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.991718 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/930df857-6e36-4c04-a098-ccec27be1dec-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:41.991899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:41.991888 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/930df857-6e36-4c04-a098-ccec27be1dec-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:42.014801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.014779 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-298px\" (UniqueName: \"kubernetes.io/projected/930df857-6e36-4c04-a098-ccec27be1dec-kube-api-access-298px\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5\" (UID: \"930df857-6e36-4c04-a098-ccec27be1dec\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:42.177995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.177917 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:42.480824 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.480801 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5"] Oct 30 05:42:42.483116 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:42:42.483090 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930df857_6e36_4c04_a098_ccec27be1dec.slice/crio-640bffba8ba22b17061b128980ea8990eb8a5b72611b2b60e1bb1ef802186574 WatchSource:0}: Error finding container 640bffba8ba22b17061b128980ea8990eb8a5b72611b2b60e1bb1ef802186574: Status 404 returned error can't find the container with id 640bffba8ba22b17061b128980ea8990eb8a5b72611b2b60e1bb1ef802186574 Oct 30 05:42:42.973097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.973057 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2"] Oct 30 05:42:42.977041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.977018 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:42.996584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.996546 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8025876f-d78e-4475-afd3-4ba65437847f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:42.996730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.996614 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:42.996730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.996644 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bwz8\" (UniqueName: \"kubernetes.io/projected/8025876f-d78e-4475-afd3-4ba65437847f-kube-api-access-8bwz8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:42.996730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.996683 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:42.996863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.996813 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:42.996915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:42.996871 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.000386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.000335 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Oct 30 05:42:43.001109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.001087 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2"] Oct 30 05:42:43.097347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097309 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8025876f-d78e-4475-afd3-4ba65437847f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.097517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097359 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.097517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097385 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bwz8\" (UniqueName: \"kubernetes.io/projected/8025876f-d78e-4475-afd3-4ba65437847f-kube-api-access-8bwz8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.097517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097414 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.097517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097446 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.097517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097485 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.097967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097939 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.098062 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.097969 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.098062 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.098032 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.099743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.099716 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8025876f-d78e-4475-afd3-4ba65437847f-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.099933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.099917 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8025876f-d78e-4475-afd3-4ba65437847f-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.128647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.128620 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bwz8\" (UniqueName: \"kubernetes.io/projected/8025876f-d78e-4475-afd3-4ba65437847f-kube-api-access-8bwz8\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2\" (UID: \"8025876f-d78e-4475-afd3-4ba65437847f\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.168184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.168153 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" event={"ID":"930df857-6e36-4c04-a098-ccec27be1dec","Type":"ContainerStarted","Data":"6d5de5acd9b4cb8c0879433aca9c255dd2a858b6f1c1b4dc458f53bec0bfe910"} Oct 30 05:42:43.168367 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.168201 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" event={"ID":"930df857-6e36-4c04-a098-ccec27be1dec","Type":"ContainerStarted","Data":"640bffba8ba22b17061b128980ea8990eb8a5b72611b2b60e1bb1ef802186574"} Oct 30 05:42:43.178649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.178629 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:43.217216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.217176 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" podStartSLOduration=2.017105857 podStartE2EDuration="2.2171636s" totalImagesPullingTime="200.057743ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:43.216016688 +0000 UTC m=+669.791703742" watchObservedRunningTime="2026-10-30 05:42:43.2171636 +0000 UTC m=+669.792850653" Oct 30 05:42:43.288592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.288571 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:43.594231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:43.594030 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2"] Oct 30 05:42:43.596766 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:42:43.596738 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8025876f_d78e_4475_afd3_4ba65437847f.slice/crio-18b214a3a0d155216ae237ddfb4ef0dcb3d3f284fca5d23dbea093b6dc85dd1e WatchSource:0}: Error finding container 18b214a3a0d155216ae237ddfb4ef0dcb3d3f284fca5d23dbea093b6dc85dd1e: Status 404 returned error can't find the container with id 18b214a3a0d155216ae237ddfb4ef0dcb3d3f284fca5d23dbea093b6dc85dd1e Oct 30 05:42:44.173664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:44.173580 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" event={"ID":"8025876f-d78e-4475-afd3-4ba65437847f","Type":"ContainerStarted","Data":"1e678b8c9a4711f873d831fc71f2cfa444ed4eff60c72e326de2c2dc2b9b8862"} Oct 30 05:42:44.173664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:44.173625 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" event={"ID":"8025876f-d78e-4475-afd3-4ba65437847f","Type":"ContainerStarted","Data":"18b214a3a0d155216ae237ddfb4ef0dcb3d3f284fca5d23dbea093b6dc85dd1e"} Oct 30 05:42:44.241854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:44.241806 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podStartSLOduration=2.03146072 podStartE2EDuration="2.24179091s" totalImagesPullingTime="210.33019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:42:44.241126554 +0000 UTC m=+670.816813608" watchObservedRunningTime="2026-10-30 05:42:44.24179091 +0000 UTC m=+670.817477964" Oct 30 05:42:44.289129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:44.289103 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:45.592653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:45.592623 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:42:47.529704 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:42:47.529668 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:42:49.226778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:49.226742 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:49.239277 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:49.239251 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:50.209003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:50.208977 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-9hmzl" Oct 30 05:42:51.110094 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:51.110060 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:42:52.179127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:52.179090 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:52.191442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:52.191417 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:52.221278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:52.221248 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5" Oct 30 05:42:53.289568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:53.289516 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:53.301844 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:53.301818 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:54.226571 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:54.226545 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:42:58.830317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.830284 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7549c48cc4-skclp"] Oct 30 05:42:58.836469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.836444 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:58.854090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.854060 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7549c48cc4-skclp"] Oct 30 05:42:58.867112 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.867084 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-pr45g\"" Oct 30 05:42:58.867112 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.867084 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 05:42:58.933957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.933922 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f-tls-cert\") pod \"authorino-7549c48cc4-skclp\" (UID: \"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f\") " pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:58.933957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:58.933962 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnh2p\" (UniqueName: \"kubernetes.io/projected/d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f-kube-api-access-gnh2p\") pod \"authorino-7549c48cc4-skclp\" (UID: \"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f\") " pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:59.034806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:59.034764 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f-tls-cert\") pod \"authorino-7549c48cc4-skclp\" (UID: \"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f\") " pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:59.034806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:59.034811 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnh2p\" (UniqueName: \"kubernetes.io/projected/d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f-kube-api-access-gnh2p\") pod \"authorino-7549c48cc4-skclp\" (UID: \"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f\") " pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:59.037354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:59.037320 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f-tls-cert\") pod \"authorino-7549c48cc4-skclp\" (UID: \"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f\") " pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:59.061766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:59.061729 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnh2p\" (UniqueName: \"kubernetes.io/projected/d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f-kube-api-access-gnh2p\") pod \"authorino-7549c48cc4-skclp\" (UID: \"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f\") " pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:59.149955 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:59.149865 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7549c48cc4-skclp" Oct 30 05:42:59.460721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:42:59.460692 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7549c48cc4-skclp"] Oct 30 05:42:59.463956 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:42:59.463928 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd32ee48f_6a9f_4c44_8a8b_f5baaf0a416f.slice/crio-813dccd32efcb61d92ebcb31266759dcaf052c62cda4b76257401aefe4211c6d WatchSource:0}: Error finding container 813dccd32efcb61d92ebcb31266759dcaf052c62cda4b76257401aefe4211c6d: Status 404 returned error can't find the container with id 813dccd32efcb61d92ebcb31266759dcaf052c62cda4b76257401aefe4211c6d Oct 30 05:43:00.242778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:00.242721 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7549c48cc4-skclp" event={"ID":"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f","Type":"ContainerStarted","Data":"c04f95783e88042cfebf6e87304d2528709d7e79cca77aa8bd02750e482e697a"} Oct 30 05:43:00.242778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:00.242775 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7549c48cc4-skclp" event={"ID":"d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f","Type":"ContainerStarted","Data":"813dccd32efcb61d92ebcb31266759dcaf052c62cda4b76257401aefe4211c6d"} Oct 30 05:43:00.285938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:00.285829 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-7549c48cc4-skclp" podStartSLOduration=1.74782224 podStartE2EDuration="2.285816903s" totalImagesPullingTime="537.994663ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:42:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:00.285606283 +0000 UTC m=+686.861293349" watchObservedRunningTime="2026-10-30 05:43:00.285816903 +0000 UTC m=+686.861503957" Oct 30 05:43:02.737214 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.737172 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6"] Oct 30 05:43:02.740043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.740022 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.761280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.761249 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 05:43:02.776738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.776694 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6"] Oct 30 05:43:02.841706 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.841670 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-77wcn\"" Oct 30 05:43:02.879160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879107 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879200 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879242 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879300 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879419 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879473 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879521 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879628 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnzgr\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.879714 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.879657 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.980818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980780 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.980818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980820 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980839 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980859 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980884 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980936 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnzgr\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.980964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.981080 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.981146 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.981299 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.981347 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.981395 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.981680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.981574 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.982068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.982042 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.983339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.983308 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:02.983815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:02.983788 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:03.013223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:03.013175 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:03.013749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:03.013729 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnzgr\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr\") pod \"maas-default-gateway-openshift-default-84587756dc-rlkc6\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:03.053511 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:03.053471 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:03.390384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:03.390354 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6"] Oct 30 05:43:03.392809 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:43:03.392785 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode12372fc_9176_4c18_885f_d77075d6f514.slice/crio-11c37270dff4a1bdbe976e57d71d05ed3cd7c031ee8e6c66a956080fad115eda WatchSource:0}: Error finding container 11c37270dff4a1bdbe976e57d71d05ed3cd7c031ee8e6c66a956080fad115eda: Status 404 returned error can't find the container with id 11c37270dff4a1bdbe976e57d71d05ed3cd7c031ee8e6c66a956080fad115eda Oct 30 05:43:04.109774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.109737 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-m62wv"] Oct 30 05:43:04.112767 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.112748 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.134106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.134081 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 05:43:04.149607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.149579 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-m62wv"] Oct 30 05:43:04.200841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.200763 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b17179d-de10-4330-a48a-e5aa6ecca579-plugins-config-volume\") pod \"payload-processing-5f78687d5b-m62wv\" (UID: \"3b17179d-de10-4330-a48a-e5aa6ecca579\") " pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.201117 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.200870 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b776x\" (UniqueName: \"kubernetes.io/projected/3b17179d-de10-4330-a48a-e5aa6ecca579-kube-api-access-b776x\") pod \"payload-processing-5f78687d5b-m62wv\" (UID: \"3b17179d-de10-4330-a48a-e5aa6ecca579\") " pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.264097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.263935 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" event={"ID":"e12372fc-9176-4c18-885f-d77075d6f514","Type":"ContainerStarted","Data":"11c37270dff4a1bdbe976e57d71d05ed3cd7c031ee8e6c66a956080fad115eda"} Oct 30 05:43:04.301668 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.301627 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b17179d-de10-4330-a48a-e5aa6ecca579-plugins-config-volume\") pod \"payload-processing-5f78687d5b-m62wv\" (UID: \"3b17179d-de10-4330-a48a-e5aa6ecca579\") " pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.301849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.301722 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b776x\" (UniqueName: \"kubernetes.io/projected/3b17179d-de10-4330-a48a-e5aa6ecca579-kube-api-access-b776x\") pod \"payload-processing-5f78687d5b-m62wv\" (UID: \"3b17179d-de10-4330-a48a-e5aa6ecca579\") " pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.302602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.302571 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b17179d-de10-4330-a48a-e5aa6ecca579-plugins-config-volume\") pod \"payload-processing-5f78687d5b-m62wv\" (UID: \"3b17179d-de10-4330-a48a-e5aa6ecca579\") " pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.327146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.327119 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b776x\" (UniqueName: \"kubernetes.io/projected/3b17179d-de10-4330-a48a-e5aa6ecca579-kube-api-access-b776x\") pod \"payload-processing-5f78687d5b-m62wv\" (UID: \"3b17179d-de10-4330-a48a-e5aa6ecca579\") " pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.424865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.424775 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:04.746595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:04.746502 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5f78687d5b-m62wv"] Oct 30 05:43:04.750883 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:43:04.750848 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b17179d_de10_4330_a48a_e5aa6ecca579.slice/crio-d638d30536c8733ed6245e24f49d14da27995befd0af1f29caa0ccefcad261d4 WatchSource:0}: Error finding container d638d30536c8733ed6245e24f49d14da27995befd0af1f29caa0ccefcad261d4: Status 404 returned error can't find the container with id d638d30536c8733ed6245e24f49d14da27995befd0af1f29caa0ccefcad261d4 Oct 30 05:43:05.271890 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:05.271851 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" event={"ID":"3b17179d-de10-4330-a48a-e5aa6ecca579","Type":"ContainerStarted","Data":"d638d30536c8733ed6245e24f49d14da27995befd0af1f29caa0ccefcad261d4"} Oct 30 05:43:06.765498 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:06.765465 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:43:06.765834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:06.765562 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:43:06.765834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:06.765612 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:43:07.283322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:07.283284 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" event={"ID":"3b17179d-de10-4330-a48a-e5aa6ecca579","Type":"ContainerStarted","Data":"ebdec1d576dbca2472acb93422012080e272a75f925c9df6c4b322894bf58ab9"} Oct 30 05:43:07.283505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:07.283374 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:07.284978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:07.284954 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" event={"ID":"e12372fc-9176-4c18-885f-d77075d6f514","Type":"ContainerStarted","Data":"6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1"} Oct 30 05:43:07.329295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:07.329253 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" podStartSLOduration=1.180973872 podStartE2EDuration="3.329242304s" totalImagesPullingTime="2.148268432s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:07.327713477 +0000 UTC m=+693.903400532" watchObservedRunningTime="2026-10-30 05:43:07.329242304 +0000 UTC m=+693.904929355" Oct 30 05:43:07.379422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:07.379369 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podStartSLOduration=2.010093098 podStartE2EDuration="5.379354805s" totalImagesPullingTime="3.369261707s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:07.378347002 +0000 UTC m=+693.954034056" watchObservedRunningTime="2026-10-30 05:43:07.379354805 +0000 UTC m=+693.955041858" Oct 30 05:43:08.077706 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:08.077665 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:08.079250 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:08.079224 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:08.079374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:08.079275 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:09.062921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:09.062884 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:09.063202 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:09.063173 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:10.077686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:10.077646 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:10.078142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:10.077708 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:11.068551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:11.061880 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:11.068551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:11.061942 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:12.063922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:12.063887 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:12.064646 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:12.064615 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:13.054051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:13.054017 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:13.054224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:13.054066 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:14.054896 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.054860 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-rlkc6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" start-of-body= Oct 30 05:43:14.055293 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.054910 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.29:15021/healthz/ready\": dial tcp 10.134.0.29:15021: connect: connection refused" Oct 30 05:43:14.337802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.337725 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-554787856d-zdmnb"] Oct 30 05:43:14.341411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.341394 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.362221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.362195 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-554787856d-zdmnb"] Oct 30 05:43:14.374758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.374737 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 05:43:14.374897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.374850 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-gqm89\"" Oct 30 05:43:14.374967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.374916 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 05:43:14.514499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.514461 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca8a08-a9dc-45b8-8331-4751c3bc20b7-metrics-certs\") pod \"ai-gateway-controller-554787856d-zdmnb\" (UID: \"6eca8a08-a9dc-45b8-8331-4751c3bc20b7\") " pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.514669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.514580 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jm52\" (UniqueName: \"kubernetes.io/projected/6eca8a08-a9dc-45b8-8331-4751c3bc20b7-kube-api-access-7jm52\") pod \"ai-gateway-controller-554787856d-zdmnb\" (UID: \"6eca8a08-a9dc-45b8-8331-4751c3bc20b7\") " pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.615016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.614932 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jm52\" (UniqueName: \"kubernetes.io/projected/6eca8a08-a9dc-45b8-8331-4751c3bc20b7-kube-api-access-7jm52\") pod \"ai-gateway-controller-554787856d-zdmnb\" (UID: \"6eca8a08-a9dc-45b8-8331-4751c3bc20b7\") " pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.615016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.614993 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca8a08-a9dc-45b8-8331-4751c3bc20b7-metrics-certs\") pod \"ai-gateway-controller-554787856d-zdmnb\" (UID: \"6eca8a08-a9dc-45b8-8331-4751c3bc20b7\") " pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.617322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.617292 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eca8a08-a9dc-45b8-8331-4751c3bc20b7-metrics-certs\") pod \"ai-gateway-controller-554787856d-zdmnb\" (UID: \"6eca8a08-a9dc-45b8-8331-4751c3bc20b7\") " pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.640221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.640192 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jm52\" (UniqueName: \"kubernetes.io/projected/6eca8a08-a9dc-45b8-8331-4751c3bc20b7-kube-api-access-7jm52\") pod \"ai-gateway-controller-554787856d-zdmnb\" (UID: \"6eca8a08-a9dc-45b8-8331-4751c3bc20b7\") " pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.657802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.657773 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:14.958589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:14.958360 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-554787856d-zdmnb"] Oct 30 05:43:14.962067 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:43:14.962040 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eca8a08_a9dc_45b8_8331_4751c3bc20b7.slice/crio-d55084973c2133313e825f5edc3a6d694afe56f734b72972eae13231ba1d2284 WatchSource:0}: Error finding container d55084973c2133313e825f5edc3a6d694afe56f734b72972eae13231ba1d2284: Status 404 returned error can't find the container with id d55084973c2133313e825f5edc3a6d694afe56f734b72972eae13231ba1d2284 Oct 30 05:43:15.072020 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:15.071991 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:15.072552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:15.072511 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:15.073402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:15.073380 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 05:43:15.321686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:15.321655 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" event={"ID":"6eca8a08-a9dc-45b8-8331-4751c3bc20b7","Type":"ContainerStarted","Data":"d55084973c2133313e825f5edc3a6d694afe56f734b72972eae13231ba1d2284"} Oct 30 05:43:17.333541 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.333496 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" event={"ID":"6eca8a08-a9dc-45b8-8331-4751c3bc20b7","Type":"ContainerStarted","Data":"4ce532ec05ddf0de019a4f06d3fa4d85fa7b6517b0a0c96cdcfcbd95ad477db4"} Oct 30 05:43:17.333921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.333623 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:17.379817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.379727 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" podStartSLOduration=1.232214376 podStartE2EDuration="3.379713806s" totalImagesPullingTime="2.14749943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:17.378586827 +0000 UTC m=+703.954273881" watchObservedRunningTime="2026-10-30 05:43:17.379713806 +0000 UTC m=+703.955400921" Oct 30 05:43:17.530596 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:43:17.530562 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:43:17.763306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.763274 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr"] Oct 30 05:43:17.766830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.766807 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.799957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.799932 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr"] Oct 30 05:43:17.948447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948420 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948455 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948476 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948547 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948629 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948662 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948745 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8611d5b8-cd8b-4117-b571-b96f2cde2621-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948766 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l5jd\" (UniqueName: \"kubernetes.io/projected/8611d5b8-cd8b-4117-b571-b96f2cde2621-kube-api-access-7l5jd\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:17.948840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:17.948825 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049625 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049664 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049685 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049713 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049732 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049760 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049814 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049870 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8611d5b8-cd8b-4117-b571-b96f2cde2621-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.049940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.049901 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l5jd\" (UniqueName: \"kubernetes.io/projected/8611d5b8-cd8b-4117-b571-b96f2cde2621-kube-api-access-7l5jd\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.050226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.050085 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.050226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.050141 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.050226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.050161 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.050369 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.050261 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.050736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.050717 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8611d5b8-cd8b-4117-b571-b96f2cde2621-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.052044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.052027 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.052159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.052142 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.075391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.075364 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8611d5b8-cd8b-4117-b571-b96f2cde2621-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.075497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.075479 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l5jd\" (UniqueName: \"kubernetes.io/projected/8611d5b8-cd8b-4117-b571-b96f2cde2621-kube-api-access-7l5jd\") pod \"maas-default-gateway-openshift-default-84587756dc-xp2kr\" (UID: \"8611d5b8-cd8b-4117-b571-b96f2cde2621\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.080277 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.080262 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:18.291285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.291258 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5f78687d5b-m62wv" Oct 30 05:43:18.400045 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.400019 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr"] Oct 30 05:43:18.402434 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:43:18.402402 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8611d5b8_cd8b_4117_b571_b96f2cde2621.slice/crio-2a20425d1d40ac8e2790da93a66eb92e1dc8d59706528664ac900dbcce138a89 WatchSource:0}: Error finding container 2a20425d1d40ac8e2790da93a66eb92e1dc8d59706528664ac900dbcce138a89: Status 404 returned error can't find the container with id 2a20425d1d40ac8e2790da93a66eb92e1dc8d59706528664ac900dbcce138a89 Oct 30 05:43:18.405410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.405380 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:43:18.405522 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.405437 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:43:18.405522 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:18.405462 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:43:19.345897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:19.345864 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" event={"ID":"8611d5b8-cd8b-4117-b571-b96f2cde2621","Type":"ContainerStarted","Data":"4f727dc9eb9341d999d602f40c03767986bd9c31712147a71b6c4a9efb854cf8"} Oct 30 05:43:19.346076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:19.345911 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" event={"ID":"8611d5b8-cd8b-4117-b571-b96f2cde2621","Type":"ContainerStarted","Data":"2a20425d1d40ac8e2790da93a66eb92e1dc8d59706528664ac900dbcce138a89"} Oct 30 05:43:19.424343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:19.424295 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podStartSLOduration=2.424282696 podStartE2EDuration="2.424282696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:43:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:43:19.410095298 +0000 UTC m=+705.985782352" watchObservedRunningTime="2026-10-30 05:43:19.424282696 +0000 UTC m=+705.999969749" Oct 30 05:43:20.080902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:20.080857 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:20.082140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:20.082104 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 05:43:20.082286 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:20.082154 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 05:43:21.081361 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:21.081326 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 05:43:21.081789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:21.081378 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 05:43:22.081065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:22.081030 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 05:43:22.081271 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:22.081083 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 05:43:23.081082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:23.081044 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 05:43:23.081521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:23.081101 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 05:43:24.081518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:24.081485 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 05:43:24.081974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:24.081561 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 05:43:25.080757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:25.080723 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body= Oct 30 05:43:25.080936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:25.080771 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" Oct 30 05:43:26.628592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:26.628562 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:26.629092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:26.628930 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:26.629768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:26.629749 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" Oct 30 05:43:28.342626 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:43:28.342597 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-554787856d-zdmnb" Oct 30 05:43:47.531990 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:43:47.531913 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:44:00.749521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.749490 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7"] Oct 30 05:44:00.753910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.753885 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.772681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.772657 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-64123f-openshift-default-dockercfg-msqjh\"" Oct 30 05:44:00.787953 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.787929 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7"] Oct 30 05:44:00.940627 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940591 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.940781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940651 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.940781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940763 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.940863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940791 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.940863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940826 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.940933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940881 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.940987 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.940934 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.941028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.941004 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqsnr\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:00.941074 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:00.941041 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042185 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042255 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042307 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042352 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqsnr\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042396 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042420 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042471 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042552 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042584 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042664 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.042884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042853 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.043012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.042965 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.043097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.043079 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.043284 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.043261 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.044621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.044597 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.044760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.044745 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.073607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.073580 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.073847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.073830 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqsnr\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.319754 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.319682 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr"] Oct 30 05:44:01.324583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.324561 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.345352 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.345323 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-fffaa8-openshift-default-dockercfg-fhmlj\"" Oct 30 05:44:01.350920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.350899 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr"] Oct 30 05:44:01.367922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.367905 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:01.446385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446348 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446398 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446435 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446514 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446566 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446617 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w488\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446649 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446674 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.446822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.446706 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.520802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.520768 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd"] Oct 30 05:44:01.524858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.524833 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.547383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547359 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547389 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547411 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547455 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547476 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547499 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w488\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547540 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547569 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547605 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547790 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.547903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.547852 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.548054 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.548026 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.548126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.548106 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.548268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.548242 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.549652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.549627 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.549943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.549928 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.557896 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.557875 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd"] Oct 30 05:44:01.572898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.572853 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-be78f2-openshift-default-dockercfg-8twx9\"" Oct 30 05:44:01.588836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.588809 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.589015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.588998 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w488\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.637855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.637835 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:01.648785 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.648764 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.648853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.648808 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2mv6\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.648918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.648902 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.648960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.648932 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.649012 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.648973 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.649110 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.649089 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.649167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.649125 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.649204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.649181 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.649243 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.649210 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.694599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.693895 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7"] Oct 30 05:44:01.694599 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:01.694209 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb26ccd6a_916a_4f95_aa97_91dfe6e3c889.slice/crio-b3ca18681b34d03e5b4aa0710b0e00698dcec837d18987106d158e3eb3de83d1 WatchSource:0}: Error finding container b3ca18681b34d03e5b4aa0710b0e00698dcec837d18987106d158e3eb3de83d1: Status 404 returned error can't find the container with id b3ca18681b34d03e5b4aa0710b0e00698dcec837d18987106d158e3eb3de83d1 Oct 30 05:44:01.697438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.697033 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:01.697438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.697116 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:01.697438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.697157 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:01.750595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750569 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750609 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2mv6\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750654 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750675 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750848 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750908 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750933 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.750986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.750967 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.751788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.751000 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.751788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.751062 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.751788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.751108 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.751788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.751450 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.751788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.751768 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.752046 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.751931 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.753386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.753362 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.753697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.753678 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.778357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.778332 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2mv6\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.779733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.779710 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.838711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.838622 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:01.969049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.969021 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr"] Oct 30 05:44:01.971548 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:01.971489 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b56a291_a225_48d1_8af8_f331c2eb4ae4.slice/crio-b5d1a58480e72d2b838a58cf1190014822f733cadcdd8d8a3bcd8f0e544e2464 WatchSource:0}: Error finding container b5d1a58480e72d2b838a58cf1190014822f733cadcdd8d8a3bcd8f0e544e2464: Status 404 returned error can't find the container with id b5d1a58480e72d2b838a58cf1190014822f733cadcdd8d8a3bcd8f0e544e2464 Oct 30 05:44:01.974524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.974488 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:01.974649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.974627 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:01.974687 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:01.974676 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:02.146315 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.146229 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd"] Oct 30 05:44:02.150119 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:02.150086 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7185eafe_7e10_483b_a8c6_afd9ea1dbf56.slice/crio-15bcd6c2e3e92269608221b1efcc40b946071c186335c809e1c8a0b9ec99b0a0 WatchSource:0}: Error finding container 15bcd6c2e3e92269608221b1efcc40b946071c186335c809e1c8a0b9ec99b0a0: Status 404 returned error can't find the container with id 15bcd6c2e3e92269608221b1efcc40b946071c186335c809e1c8a0b9ec99b0a0 Oct 30 05:44:02.153096 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.153056 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:02.153206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.153143 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:02.153206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.153186 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:02.560772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.560738 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" event={"ID":"7185eafe-7e10-483b-a8c6-afd9ea1dbf56","Type":"ContainerStarted","Data":"bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149"} Oct 30 05:44:02.560973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.560791 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" event={"ID":"7185eafe-7e10-483b-a8c6-afd9ea1dbf56","Type":"ContainerStarted","Data":"15bcd6c2e3e92269608221b1efcc40b946071c186335c809e1c8a0b9ec99b0a0"} Oct 30 05:44:02.562265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.562237 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" event={"ID":"9b56a291-a225-48d1-8af8-f331c2eb4ae4","Type":"ContainerStarted","Data":"8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954"} Oct 30 05:44:02.562399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.562283 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" event={"ID":"9b56a291-a225-48d1-8af8-f331c2eb4ae4","Type":"ContainerStarted","Data":"b5d1a58480e72d2b838a58cf1190014822f733cadcdd8d8a3bcd8f0e544e2464"} Oct 30 05:44:02.563674 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.563648 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" event={"ID":"b26ccd6a-916a-4f95-aa97-91dfe6e3c889","Type":"ContainerStarted","Data":"480564cdd24754f2d1cc269989c26c006458b996998983e0fc0a368254a5039c"} Oct 30 05:44:02.563778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.563690 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" event={"ID":"b26ccd6a-916a-4f95-aa97-91dfe6e3c889","Type":"ContainerStarted","Data":"b3ca18681b34d03e5b4aa0710b0e00698dcec837d18987106d158e3eb3de83d1"} Oct 30 05:44:02.622456 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.622404 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podStartSLOduration=1.622391513 podStartE2EDuration="1.622391513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:02.621806377 +0000 UTC m=+749.197493431" watchObservedRunningTime="2026-10-30 05:44:02.622391513 +0000 UTC m=+749.198078567" Oct 30 05:44:02.638684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.638656 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:02.675985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.675945 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podStartSLOduration=2.6759322 podStartE2EDuration="2.6759322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:02.675295947 +0000 UTC m=+749.250983002" watchObservedRunningTime="2026-10-30 05:44:02.6759322 +0000 UTC m=+749.251619253" Oct 30 05:44:02.739503 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.739457 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podStartSLOduration=1.739445932 podStartE2EDuration="1.739445932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:02.739208125 +0000 UTC m=+749.314895178" watchObservedRunningTime="2026-10-30 05:44:02.739445932 +0000 UTC m=+749.315132986" Oct 30 05:44:02.839771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:02.839714 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:03.368131 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:03.368093 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:03.373021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:03.372998 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:03.567863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:03.567831 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:03.568618 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:03.568598 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:03.644477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:03.644396 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:03.844430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:03.844402 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:04.572915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:04.572879 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:04.573115 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:04.572987 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:04.573639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:04.573620 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:04.573808 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:04.573790 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:07.124840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.124807 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5"] Oct 30 05:44:07.128616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.128596 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.169741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.169716 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-698ab2-dockercfg-bt94d\"" Oct 30 05:44:07.169915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.169901 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-698ab2\"" Oct 30 05:44:07.172379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.172357 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5"] Oct 30 05:44:07.199399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.199379 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b"] Oct 30 05:44:07.202008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.201986 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.202188 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.202167 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dshtd\" (UniqueName: \"kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd\") pod \"payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.202761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.202747 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.225485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.225464 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b"] Oct 30 05:44:07.303376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.303343 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.303587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.303420 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcr8b\" (UniqueName: \"kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b\") pod \"payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.303587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.303507 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dshtd\" (UniqueName: \"kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd\") pod \"payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.303587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.303569 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.304274 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.304256 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.339455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.339426 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dshtd\" (UniqueName: \"kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd\") pod \"payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.404823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.404763 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.404823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.404811 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcr8b\" (UniqueName: \"kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b\") pod \"payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.405414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.405396 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.433866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.433842 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcr8b\" (UniqueName: \"kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b\") pod \"payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.441733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.441711 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:07.515926 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.515854 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:07.749412 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.749391 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5"] Oct 30 05:44:07.750912 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:07.750885 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ac71497_b80b_4ab7_a3f1_c84cafc330c3.slice/crio-24f02afe386b0463b65b6e265b2f9ba2a7bf2738ebd1c4bce95305f2dd0baa17 WatchSource:0}: Error finding container 24f02afe386b0463b65b6e265b2f9ba2a7bf2738ebd1c4bce95305f2dd0baa17: Status 404 returned error can't find the container with id 24f02afe386b0463b65b6e265b2f9ba2a7bf2738ebd1c4bce95305f2dd0baa17 Oct 30 05:44:07.821363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:07.821341 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b"] Oct 30 05:44:07.823138 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:07.823114 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3984d473_1aad_4f91_8268_24e38f11cf79.slice/crio-622e2fe362486742928a93f9bc0496c86396ba4e62f2084d41b952f54a4224be WatchSource:0}: Error finding container 622e2fe362486742928a93f9bc0496c86396ba4e62f2084d41b952f54a4224be: Status 404 returned error can't find the container with id 622e2fe362486742928a93f9bc0496c86396ba4e62f2084d41b952f54a4224be Oct 30 05:44:08.591724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.591690 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" event={"ID":"3984d473-1aad-4f91-8268-24e38f11cf79","Type":"ContainerStarted","Data":"70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1"} Oct 30 05:44:08.592184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.591742 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:08.592184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.591767 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" event={"ID":"3984d473-1aad-4f91-8268-24e38f11cf79","Type":"ContainerStarted","Data":"622e2fe362486742928a93f9bc0496c86396ba4e62f2084d41b952f54a4224be"} Oct 30 05:44:08.593272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.593248 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" event={"ID":"1ac71497-b80b-4ab7-a3f1-c84cafc330c3","Type":"ContainerStarted","Data":"030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe"} Oct 30 05:44:08.593367 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.593293 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:08.593367 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.593308 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" event={"ID":"1ac71497-b80b-4ab7-a3f1-c84cafc330c3","Type":"ContainerStarted","Data":"24f02afe386b0463b65b6e265b2f9ba2a7bf2738ebd1c4bce95305f2dd0baa17"} Oct 30 05:44:08.646645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.646595 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" podStartSLOduration=1.646582317 podStartE2EDuration="1.646582317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:08.644449488 +0000 UTC m=+755.220136554" watchObservedRunningTime="2026-10-30 05:44:08.646582317 +0000 UTC m=+755.222269371" Oct 30 05:44:08.690627 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.690575 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" podStartSLOduration=1.690560217 podStartE2EDuration="1.690560217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:08.689768018 +0000 UTC m=+755.265455071" watchObservedRunningTime="2026-10-30 05:44:08.690560217 +0000 UTC m=+755.266247272" Oct 30 05:44:08.984574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.984486 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd"] Oct 30 05:44:08.988185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:08.988167 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.016661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.016641 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-1d8c2f\"" Oct 30 05:44:09.022033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.022012 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.022113 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.022080 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-stnqj\" (UniqueName: \"kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj\") pod \"payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.030218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.030200 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd"] Oct 30 05:44:09.040926 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.040907 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds"] Oct 30 05:44:09.044240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.044218 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.063749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.063724 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-1d8c2f-dockercfg-vvqqr\"" Oct 30 05:44:09.075406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.075381 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds"] Oct 30 05:44:09.123637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.123607 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.123767 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.123715 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7bbb\" (UniqueName: \"kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb\") pod \"payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.123862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.123841 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-stnqj\" (UniqueName: \"kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj\") pod \"payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.123983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.123967 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.124627 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.124607 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.164231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.164189 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-stnqj\" (UniqueName: \"kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj\") pod \"payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.224850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.224817 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.225001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.224866 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7bbb\" (UniqueName: \"kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb\") pod \"payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.225506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.225487 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.261995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.261966 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7bbb\" (UniqueName: \"kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb\") pod \"payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.300826 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.300801 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:09.358939 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.358908 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:09.606278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.606251 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd"] Oct 30 05:44:09.607987 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:09.607960 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e26684e_e7b1_4118_b69e_c5375a0586b3.slice/crio-3f04994b9fcee97b14e8c9c941abeb5b4e3a5a658f6fcb3db77feb263b4f9331 WatchSource:0}: Error finding container 3f04994b9fcee97b14e8c9c941abeb5b4e3a5a658f6fcb3db77feb263b4f9331: Status 404 returned error can't find the container with id 3f04994b9fcee97b14e8c9c941abeb5b4e3a5a658f6fcb3db77feb263b4f9331 Oct 30 05:44:09.676060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.676039 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds"] Oct 30 05:44:09.678107 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:09.678084 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc496ad6_4aea_4eb6_acc0_001f579881bf.slice/crio-5e6ffbe1eaa4444cbd8c0b91d6a0002f4a41cf63469f06d12042db03bf5d990c WatchSource:0}: Error finding container 5e6ffbe1eaa4444cbd8c0b91d6a0002f4a41cf63469f06d12042db03bf5d990c: Status 404 returned error can't find the container with id 5e6ffbe1eaa4444cbd8c0b91d6a0002f4a41cf63469f06d12042db03bf5d990c Oct 30 05:44:09.941622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.941552 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl"] Oct 30 05:44:09.945620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.945603 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:09.972947 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.972924 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-64123f\"" Oct 30 05:44:09.975081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:09.975053 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl"] Oct 30 05:44:10.033201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.033173 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfxw\" (UniqueName: \"kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw\") pod \"payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.033322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.033227 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.134727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.134692 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfxw\" (UniqueName: \"kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw\") pod \"payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.134912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.134786 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.135431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.135409 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.160474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.160450 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfxw\" (UniqueName: \"kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw\") pod \"payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.257452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.257427 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:10.555632 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.555552 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl"] Oct 30 05:44:10.557981 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:10.557949 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod045d509b_c8d2_4e2b_a687_01058f2097a4.slice/crio-e8fd169139269ad1d13c5bea091734360272ad3279cf05abd287a6f4dc88419f WatchSource:0}: Error finding container e8fd169139269ad1d13c5bea091734360272ad3279cf05abd287a6f4dc88419f: Status 404 returned error can't find the container with id e8fd169139269ad1d13c5bea091734360272ad3279cf05abd287a6f4dc88419f Oct 30 05:44:10.607028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.606996 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" event={"ID":"cc496ad6-4aea-4eb6-acc0-001f579881bf","Type":"ContainerStarted","Data":"ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13"} Oct 30 05:44:10.607390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.607057 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" event={"ID":"cc496ad6-4aea-4eb6-acc0-001f579881bf","Type":"ContainerStarted","Data":"5e6ffbe1eaa4444cbd8c0b91d6a0002f4a41cf63469f06d12042db03bf5d990c"} Oct 30 05:44:10.607390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.607107 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:10.609421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.609396 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" event={"ID":"6e26684e-e7b1-4118-b69e-c5375a0586b3","Type":"ContainerStarted","Data":"9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9"} Oct 30 05:44:10.609633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.609464 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" event={"ID":"6e26684e-e7b1-4118-b69e-c5375a0586b3","Type":"ContainerStarted","Data":"3f04994b9fcee97b14e8c9c941abeb5b4e3a5a658f6fcb3db77feb263b4f9331"} Oct 30 05:44:10.609633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.609500 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:10.611325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.611297 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" event={"ID":"045d509b-c8d2-4e2b-a687-01058f2097a4","Type":"ContainerStarted","Data":"e8fd169139269ad1d13c5bea091734360272ad3279cf05abd287a6f4dc88419f"} Oct 30 05:44:10.652602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.652560 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" podStartSLOduration=1.65254667 podStartE2EDuration="1.65254667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:10.651870521 +0000 UTC m=+757.227557576" watchObservedRunningTime="2026-10-30 05:44:10.65254667 +0000 UTC m=+757.228233725" Oct 30 05:44:10.699079 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.699024 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" podStartSLOduration=2.6990057309999997 podStartE2EDuration="2.699005731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:10.697277815 +0000 UTC m=+757.272964868" watchObservedRunningTime="2026-10-30 05:44:10.699005731 +0000 UTC m=+757.274692788" Oct 30 05:44:10.814709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:10.814636 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:44:11.618850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:11.618811 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" event={"ID":"045d509b-c8d2-4e2b-a687-01058f2097a4","Type":"ContainerStarted","Data":"1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff"} Oct 30 05:44:11.619292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:11.619234 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:11.674315 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:11.674273 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" podStartSLOduration=2.6742603799999998 podStartE2EDuration="2.67426038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:11.67220178 +0000 UTC m=+758.247888834" watchObservedRunningTime="2026-10-30 05:44:11.67426038 +0000 UTC m=+758.249947464" Oct 30 05:44:12.425367 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.425338 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd"] Oct 30 05:44:12.429095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.429079 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.457759 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.457735 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xpjh\" (UniqueName: \"kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh\") pod \"payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.457873 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.457797 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.465419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.465394 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-be78f2-dockercfg-khvsp\"" Oct 30 05:44:12.465516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.465471 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-be78f2\"" Oct 30 05:44:12.467130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.467109 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd"] Oct 30 05:44:12.559621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.559567 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xpjh\" (UniqueName: \"kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh\") pod \"payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.559810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.559682 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.560720 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.560665 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.585569 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.585518 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xpjh\" (UniqueName: \"kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh\") pod \"payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:12.741629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:12.741555 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:13.043812 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:13.043789 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd"] Oct 30 05:44:13.050691 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:13.050659 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc52be5c8_a2c5_4617_9d9a_09e61374cfdd.slice/crio-13b4ab62551c9c1c196eae011ed6a2998cb08f47f76affd617c995ed8b33eaec WatchSource:0}: Error finding container 13b4ab62551c9c1c196eae011ed6a2998cb08f47f76affd617c995ed8b33eaec: Status 404 returned error can't find the container with id 13b4ab62551c9c1c196eae011ed6a2998cb08f47f76affd617c995ed8b33eaec Oct 30 05:44:13.630705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:13.630673 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" event={"ID":"c52be5c8-a2c5-4617-9d9a-09e61374cfdd","Type":"ContainerStarted","Data":"051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e"} Oct 30 05:44:13.630908 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:13.630722 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" event={"ID":"c52be5c8-a2c5-4617-9d9a-09e61374cfdd","Type":"ContainerStarted","Data":"13b4ab62551c9c1c196eae011ed6a2998cb08f47f76affd617c995ed8b33eaec"} Oct 30 05:44:13.630908 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:13.630779 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:14.113557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:14.113320 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" podStartSLOduration=2.113299188 podStartE2EDuration="2.113299188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:13.679775387 +0000 UTC m=+760.255462441" watchObservedRunningTime="2026-10-30 05:44:14.113299188 +0000 UTC m=+760.688986243" Oct 30 05:44:14.113557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:14.113555 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:44:15.055230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.055196 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2"] Oct 30 05:44:15.059277 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.059252 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.089592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.089569 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2"] Oct 30 05:44:15.135935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.135899 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-a14091-dockercfg-6ns5h\"" Oct 30 05:44:15.135935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.135900 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-a14091\"" Oct 30 05:44:15.192605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.192575 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.192751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.192620 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8qgc\" (UniqueName: \"kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc\") pod \"payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.293952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.293920 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.294108 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.293960 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8qgc\" (UniqueName: \"kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc\") pod \"payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.294614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.294594 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.324910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.324850 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8qgc\" (UniqueName: \"kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc\") pod \"payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.371720 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.371694 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:15.673379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.673304 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2"] Oct 30 05:44:15.675933 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:15.675909 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f2c6831_2df4_4b85_a226_51627753114d.slice/crio-ae939468ade3fba64885cb0ff2f386d600ae16a68f0d1c1d5b26dc2b6d323f58 WatchSource:0}: Error finding container ae939468ade3fba64885cb0ff2f386d600ae16a68f0d1c1d5b26dc2b6d323f58: Status 404 returned error can't find the container with id ae939468ade3fba64885cb0ff2f386d600ae16a68f0d1c1d5b26dc2b6d323f58 Oct 30 05:44:15.929129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.929054 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v"] Oct 30 05:44:15.932843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.932822 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:15.955944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.955921 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-e7a506\"" Oct 30 05:44:15.957734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.957702 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v"] Oct 30 05:44:15.972154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.972131 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b"] Oct 30 05:44:15.975721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.975706 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:15.996799 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.996776 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b"] Oct 30 05:44:15.997665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:15.997648 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-e7a506-dockercfg-22n4b\"" Oct 30 05:44:16.001150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.001129 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.001233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.001189 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mvkv\" (UniqueName: \"kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv\") pod \"payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.102092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.102055 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbwkl\" (UniqueName: \"kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl\") pod \"payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.102237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.102221 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.102299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.102285 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mvkv\" (UniqueName: \"kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv\") pod \"payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.102360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.102343 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.103228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.103200 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.128323 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.128299 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mvkv\" (UniqueName: \"kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv\") pod \"payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.203629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.203576 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.203629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.203624 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbwkl\" (UniqueName: \"kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl\") pod \"payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.204366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.204341 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.229637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.229617 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbwkl\" (UniqueName: \"kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl\") pod \"payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.245437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.245418 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.288824 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.288790 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:16.544638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.544613 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v"] Oct 30 05:44:16.546793 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:16.546757 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715c1824_5f59_4aec_9191_a6abe7f3085b.slice/crio-dbcb6e7d4e256e5e4bdd03be4d9d6b5b5eab35ca612f2624bc7a33a0b41975c4 WatchSource:0}: Error finding container dbcb6e7d4e256e5e4bdd03be4d9d6b5b5eab35ca612f2624bc7a33a0b41975c4: Status 404 returned error can't find the container with id dbcb6e7d4e256e5e4bdd03be4d9d6b5b5eab35ca612f2624bc7a33a0b41975c4 Oct 30 05:44:16.598870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.598849 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b"] Oct 30 05:44:16.601188 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:16.601164 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f73b1dc_1476_4eb4_80fb_ab0f38b8dc7c.slice/crio-cdf077dd09848e83f4e341c392404259fd886b6260bc4775c9b55f342a16a4fe WatchSource:0}: Error finding container cdf077dd09848e83f4e341c392404259fd886b6260bc4775c9b55f342a16a4fe: Status 404 returned error can't find the container with id cdf077dd09848e83f4e341c392404259fd886b6260bc4775c9b55f342a16a4fe Oct 30 05:44:16.648916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.648881 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" event={"ID":"715c1824-5f59-4aec-9191-a6abe7f3085b","Type":"ContainerStarted","Data":"e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2"} Oct 30 05:44:16.649035 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.648946 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" event={"ID":"715c1824-5f59-4aec-9191-a6abe7f3085b","Type":"ContainerStarted","Data":"dbcb6e7d4e256e5e4bdd03be4d9d6b5b5eab35ca612f2624bc7a33a0b41975c4"} Oct 30 05:44:16.649035 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.648980 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:16.650473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.650449 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" event={"ID":"6f2c6831-2df4-4b85-a226-51627753114d","Type":"ContainerStarted","Data":"c5e731b9f2238af08360c0aaddd5a71be950f320e9dbc12db61c2ebbbcf6346b"} Oct 30 05:44:16.650591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.650494 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" event={"ID":"6f2c6831-2df4-4b85-a226-51627753114d","Type":"ContainerStarted","Data":"ae939468ade3fba64885cb0ff2f386d600ae16a68f0d1c1d5b26dc2b6d323f58"} Oct 30 05:44:16.650591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.650569 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:16.651807 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.651781 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" event={"ID":"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c","Type":"ContainerStarted","Data":"cdf077dd09848e83f4e341c392404259fd886b6260bc4775c9b55f342a16a4fe"} Oct 30 05:44:16.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.750811 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" podStartSLOduration=1.7507987630000001 podStartE2EDuration="1.750798763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:16.749973907 +0000 UTC m=+763.325660963" watchObservedRunningTime="2026-10-30 05:44:16.750798763 +0000 UTC m=+763.326485817" Oct 30 05:44:16.751274 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:16.751252 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" podStartSLOduration=1.751246513 podStartE2EDuration="1.751246513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:16.703155146 +0000 UTC m=+763.278842199" watchObservedRunningTime="2026-10-30 05:44:16.751246513 +0000 UTC m=+763.326933575" Oct 30 05:44:17.537032 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:44:17.536997 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:44:17.657578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:17.657522 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" event={"ID":"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c","Type":"ContainerStarted","Data":"087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6"} Oct 30 05:44:17.657777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:17.657761 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:17.709149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:17.709104 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" podStartSLOduration=2.709091662 podStartE2EDuration="2.709091662s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:17.706912547 +0000 UTC m=+764.282599601" watchObservedRunningTime="2026-10-30 05:44:17.709091662 +0000 UTC m=+764.284778716" Oct 30 05:44:19.601888 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:19.601861 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 05:44:19.602338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:19.601931 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 05:44:21.621977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:21.621945 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 05:44:21.622458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:21.622030 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 05:44:23.633380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:23.633349 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 05:44:24.639038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:24.639005 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 05:44:25.387955 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:25.387924 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:44:27.684707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:27.684676 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 05:44:27.685325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:27.685297 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 05:44:28.681587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.681564 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 05:44:28.799124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.799044 5565 generic.go:356] "Generic (PLEG): container finished" podID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerID="480564cdd24754f2d1cc269989c26c006458b996998983e0fc0a368254a5039c" exitCode=0 Oct 30 05:44:28.799492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.799118 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" event={"ID":"b26ccd6a-916a-4f95-aa97-91dfe6e3c889","Type":"ContainerDied","Data":"480564cdd24754f2d1cc269989c26c006458b996998983e0fc0a368254a5039c"} Oct 30 05:44:28.799492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.799462 5565 scope.go:122] "RemoveContainer" containerID="480564cdd24754f2d1cc269989c26c006458b996998983e0fc0a368254a5039c" Oct 30 05:44:28.800996 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.800971 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:28.801072 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.801033 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:28.801504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:28.801466 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:29.808465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.808431 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" event={"ID":"b26ccd6a-916a-4f95-aa97-91dfe6e3c889","Type":"ContainerStarted","Data":"21f242929e742ff787fc712c63906d0039095df46abbe57f997df5598ca6064a"} Oct 30 05:44:29.810219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.810191 5565 generic.go:356] "Generic (PLEG): container finished" podID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerID="bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149" exitCode=0 Oct 30 05:44:29.810372 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.810247 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" event={"ID":"7185eafe-7e10-483b-a8c6-afd9ea1dbf56","Type":"ContainerDied","Data":"bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149"} Oct 30 05:44:29.810574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.810555 5565 scope.go:122] "RemoveContainer" containerID="bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149" Oct 30 05:44:29.811892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.811862 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:29.812001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.811946 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:29.812001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:29.811991 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:30.370581 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:30.368663 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:30.370581 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:30.369981 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:30.370581 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:30.370044 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:30.841262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:30.840285 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" event={"ID":"7185eafe-7e10-483b-a8c6-afd9ea1dbf56","Type":"ContainerStarted","Data":"59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185"} Oct 30 05:44:31.372036 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:31.371999 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:31.372361 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:31.372318 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:31.811552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:31.809691 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:44:31.843318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:31.841156 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:31.843318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:31.842709 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:44:31.843318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:31.842761 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:44:32.369393 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:32.368920 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:32.369393 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:32.368983 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:32.839318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:32.839272 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:44:32.839504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:32.839338 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:44:33.382550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:33.380916 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:33.382550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:33.380977 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:33.651642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:33.649900 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:33.839548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:33.839415 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:44:33.839548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:33.839472 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:44:34.394563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:34.393977 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body= Oct 30 05:44:34.394563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:34.394042 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" Oct 30 05:44:34.576663 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:34.576625 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:34.839055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:34.839015 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:44:34.839228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:34.839073 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:44:35.423866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:35.423801 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:35.428129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:35.428104 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 05:44:35.624717 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:35.624682 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:44:35.624874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:35.624729 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:44:35.839329 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:35.839291 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 30 05:44:35.839505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:35.839364 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 30 05:44:36.843547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:36.843489 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:36.844638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:36.844615 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 05:44:37.900407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:37.900375 5565 generic.go:356] "Generic (PLEG): container finished" podID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerID="8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954" exitCode=0 Oct 30 05:44:37.900878 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:37.900413 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" event={"ID":"9b56a291-a225-48d1-8af8-f331c2eb4ae4","Type":"ContainerDied","Data":"8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954"} Oct 30 05:44:37.900878 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:37.900785 5565 scope.go:122] "RemoveContainer" containerID="8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954" Oct 30 05:44:37.901933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:37.901907 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:37.901994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:37.901972 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:37.902028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:37.902002 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:44:38.908003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:38.907969 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" event={"ID":"9b56a291-a225-48d1-8af8-f331c2eb4ae4","Type":"ContainerStarted","Data":"3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64"} Oct 30 05:44:39.638333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:39.638304 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:39.638572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:39.638555 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:39.638627 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:39.638601 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:40.638435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:40.638399 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:40.638871 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:40.638451 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:41.638761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:41.638728 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:41.639149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:41.638773 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:42.638281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:42.638244 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:42.638453 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:42.638297 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:43.116679 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.116644 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245"] Oct 30 05:44:43.122516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.122490 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.144666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.144643 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245"] Oct 30 05:44:43.180984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.180620 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"simulator-w6-64123f-kserve-self-signed-certs\"" Oct 30 05:44:43.180984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.180717 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"kube-root-ca.crt\"" Oct 30 05:44:43.180984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.180819 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"default-dockercfg-d9ktx\"" Oct 30 05:44:43.194810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.194785 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"openshift-service-ca.crt\"" Oct 30 05:44:43.225264 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.225237 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.225410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.225287 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.225410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.225303 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.225520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.225426 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.225520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.225457 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.225634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.225572 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5nwg\" (UniqueName: \"kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326420 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326389 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326420 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326420 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326588 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326649 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326708 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5nwg\" (UniqueName: \"kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326806 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326844 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.326903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.326868 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.327172 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.327153 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.328747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.328721 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.328988 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.328969 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.377610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.377504 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5nwg\" (UniqueName: \"kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg\") pod \"simulator-w6-64123f-kserve-579dfb4489-zr245\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.436150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.436121 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:43.638312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.638243 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body= Oct 30 05:44:43.638312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.638299 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" Oct 30 05:44:43.777552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.774708 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245"] Oct 30 05:44:43.781997 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:44:43.781975 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c38aaef_b590_4231_bb2f_d311af3e8002.slice/crio-9305c0c53d542ca7550977ff8fdeaad5f374d6528780bc67a038ca2a0e611876 WatchSource:0}: Error finding container 9305c0c53d542ca7550977ff8fdeaad5f374d6528780bc67a038ca2a0e611876: Status 404 returned error can't find the container with id 9305c0c53d542ca7550977ff8fdeaad5f374d6528780bc67a038ca2a0e611876 Oct 30 05:44:43.934366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.934283 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" event={"ID":"1c38aaef-b590-4231-bb2f-d311af3e8002","Type":"ContainerStarted","Data":"848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22"} Oct 30 05:44:43.934366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.934322 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" event={"ID":"1c38aaef-b590-4231-bb2f-d311af3e8002","Type":"ContainerStarted","Data":"9305c0c53d542ca7550977ff8fdeaad5f374d6528780bc67a038ca2a0e611876"} Oct 30 05:44:43.981379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:43.981329 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podStartSLOduration=0.981314952 podStartE2EDuration="981.314952ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:44:43.979584874 +0000 UTC m=+790.555271927" watchObservedRunningTime="2026-10-30 05:44:43.981314952 +0000 UTC m=+790.557002006" Oct 30 05:44:44.436604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:44.436574 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:44.641590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:44.641565 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:44.642100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:44.642050 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:44.642858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:44.642838 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 05:44:46.509079 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:46.509049 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:44:47.530467 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:44:47.530416 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:44:53.436600 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:53.436566 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:53.449167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:53.449142 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:44:53.995257 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:44:53.995224 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:45:00.205354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.205307 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v"] Oct 30 05:45:00.209074 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.209051 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w"] Oct 30 05:45:00.209216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.209164 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.213014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.212997 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl"] Oct 30 05:45:00.213149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.213111 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.216758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.216738 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.229899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.229875 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:45:00.230245 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.230231 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:45:00.230483 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.230467 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:45:00.236070 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.235444 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v"] Oct 30 05:45:00.248026 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.247999 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr"] Oct 30 05:45:00.252746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.252726 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv"] Oct 30 05:45:00.252851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.252831 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.264831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.264809 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w"] Oct 30 05:45:00.264923 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.264894 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.279817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.279797 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl"] Oct 30 05:45:00.286134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.286106 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr"] Oct 30 05:45:00.289106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.289079 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.289199 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.289139 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.289325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.289304 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m862r\" (UniqueName: \"kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.289394 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.289348 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nmh9\" (UniqueName: \"kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.289446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.289427 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.289554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.289517 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5qtb\" (UniqueName: \"kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.323395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.323336 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv"] Oct 30 05:45:00.390351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390325 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.390468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390361 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.390468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390381 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m862r\" (UniqueName: \"kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.390468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390400 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nmh9\" (UniqueName: \"kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.390468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390436 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.390468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390465 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd2bh\" (UniqueName: \"kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.390760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390487 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5qtb\" (UniqueName: \"kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.390760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390514 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-x4vwr\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.390760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390574 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bthkq\" (UniqueName: \"kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq\") pod \"maas-api-key-cleanup-29888985-x4vwr\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.390760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.390592 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.391102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.391079 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.391177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.391083 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.391230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.391208 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.426987 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.426965 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5qtb\" (UniqueName: \"kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.427109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.427053 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nmh9\" (UniqueName: \"kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9\") pod \"maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.427944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.427923 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m862r\" (UniqueName: \"kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r\") pod \"maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.492001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.491946 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd2bh\" (UniqueName: \"kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.492001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.491984 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-x4vwr\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.492158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.492013 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bthkq\" (UniqueName: \"kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq\") pod \"maas-api-key-cleanup-29888985-x4vwr\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.492158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.492031 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.492831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.492801 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.492958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.492936 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29888985-x4vwr\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.517233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.517205 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bthkq\" (UniqueName: \"kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq\") pod \"maas-api-key-cleanup-29888985-x4vwr\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.518056 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.518036 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd2bh\" (UniqueName: \"kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.525732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.525708 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:00.538283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.538261 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:00.552325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.552305 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:00.581001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.580182 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:00.584900 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.584648 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:00.841461 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.841438 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v"] Oct 30 05:45:00.844369 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:00.844342 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40533c19_4848_415b_b48c_8cc6fdcf431b.slice/crio-210d13384a84f5fb0a130a74039ea735cc68fd3215b8a9416d6e96035fa66ba6 WatchSource:0}: Error finding container 210d13384a84f5fb0a130a74039ea735cc68fd3215b8a9416d6e96035fa66ba6: Status 404 returned error can't find the container with id 210d13384a84f5fb0a130a74039ea735cc68fd3215b8a9416d6e96035fa66ba6 Oct 30 05:45:00.897823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.897799 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl"] Oct 30 05:45:00.899400 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:00.899376 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe3faf3a_984f_4081_a3e3_e068ef4c6e49.slice/crio-59a6ba738562a62b6cb05c77f6a35e0be178995e2429965696bc0499e3d08357 WatchSource:0}: Error finding container 59a6ba738562a62b6cb05c77f6a35e0be178995e2429965696bc0499e3d08357: Status 404 returned error can't find the container with id 59a6ba738562a62b6cb05c77f6a35e0be178995e2429965696bc0499e3d08357 Oct 30 05:45:00.899611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.899590 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w"] Oct 30 05:45:00.900346 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:00.900308 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91bcd3fb_329b_4b09_bbea_3ff20a08e742.slice/crio-ee9f7659a2fdb965697cabfb09fe8fffba321bf17096094ca78850a4ff7cb35a WatchSource:0}: Error finding container ee9f7659a2fdb965697cabfb09fe8fffba321bf17096094ca78850a4ff7cb35a: Status 404 returned error can't find the container with id ee9f7659a2fdb965697cabfb09fe8fffba321bf17096094ca78850a4ff7cb35a Oct 30 05:45:00.946817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.946795 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr"] Oct 30 05:45:00.948376 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:00.948348 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode574cb7e_b0a8_42be_b7d0_08e3ed4079dc.slice/crio-6bc70983701206a98664e136bafd721b082095b422f837bef2d499d0c89b0276 WatchSource:0}: Error finding container 6bc70983701206a98664e136bafd721b082095b422f837bef2d499d0c89b0276: Status 404 returned error can't find the container with id 6bc70983701206a98664e136bafd721b082095b422f837bef2d499d0c89b0276 Oct 30 05:45:00.949082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:00.949058 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv"] Oct 30 05:45:00.949452 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:00.949429 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3e78dd0_9242_48b7_b4ca_9c43a97bc4a2.slice/crio-3bb56d4883a66257157c89754d0d628f9bacf5db0a86a65ce9bd25002dd94876 WatchSource:0}: Error finding container 3bb56d4883a66257157c89754d0d628f9bacf5db0a86a65ce9bd25002dd94876: Status 404 returned error can't find the container with id 3bb56d4883a66257157c89754d0d628f9bacf5db0a86a65ce9bd25002dd94876 Oct 30 05:45:01.021684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:01.021660 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerStarted","Data":"ee9f7659a2fdb965697cabfb09fe8fffba321bf17096094ca78850a4ff7cb35a"} Oct 30 05:45:01.022862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:01.022840 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerStarted","Data":"59a6ba738562a62b6cb05c77f6a35e0be178995e2429965696bc0499e3d08357"} Oct 30 05:45:01.023949 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:01.023927 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerStarted","Data":"3bb56d4883a66257157c89754d0d628f9bacf5db0a86a65ce9bd25002dd94876"} Oct 30 05:45:01.025176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:01.025156 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerStarted","Data":"6bc70983701206a98664e136bafd721b082095b422f837bef2d499d0c89b0276"} Oct 30 05:45:01.026306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:01.026287 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerStarted","Data":"210d13384a84f5fb0a130a74039ea735cc68fd3215b8a9416d6e96035fa66ba6"} Oct 30 05:45:04.247771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.247742 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:45:04.861364 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.861334 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d"] Oct 30 05:45:04.865411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.865393 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:04.900020 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.899987 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d"] Oct 30 05:45:04.911347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.910771 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"kube-root-ca.crt\"" Oct 30 05:45:04.911347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.910915 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"openshift-service-ca.crt\"" Oct 30 05:45:04.911347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.911009 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"simulator-w0-a14091-kserve-self-signed-certs\"" Oct 30 05:45:04.911671 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.911642 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-a14091\"/\"default-dockercfg-c4xwk\"" Oct 30 05:45:04.952977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.952942 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:04.953138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.953008 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:04.953138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.953110 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:04.953267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.953168 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:04.953267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.953207 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:04.953376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:04.953305 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf26c\" (UniqueName: \"kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.056073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.055273 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.056073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.055333 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.056073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.055421 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf26c\" (UniqueName: \"kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.056073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.055459 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.056073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.055498 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.056073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.055627 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.058005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.057706 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.058005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.057958 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.058538 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.058497 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.065392 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.065339 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.069278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.068373 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerStarted","Data":"d089b4fd18201320484d7ae323e67aad2aa0cbc942fc156b5a888b7c486119d1"} Oct 30 05:45:05.071405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.071375 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerStarted","Data":"a07aac3ef75573ecb9982552d53f60e77815f9b2cf6f4b4662d12a5d43e2f35b"} Oct 30 05:45:05.072921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.072882 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.075423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.075393 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerStarted","Data":"4356a4befc023a217c023bfc6da89e3c892b539d5712f45745bb48e1d903e56e"} Oct 30 05:45:05.078197 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.077623 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerStarted","Data":"5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246"} Oct 30 05:45:05.078197 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.077915 5565 scope.go:122] "RemoveContainer" containerID="5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246" Oct 30 05:45:05.083382 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.082276 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerStarted","Data":"825bd93914bf52c91cefa843bf81595d4e4571a2d62a499a8cec8d20cfc91f11"} Oct 30 05:45:05.090393 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.090366 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf26c\" (UniqueName: \"kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c\") pod \"simulator-w0-a14091-kserve-5dc675cd75-qdt4d\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.121514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.121469 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" podStartSLOduration=1.959771798 podStartE2EDuration="5.121456361s" totalImagesPullingTime="3.161684563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:05.118638782 +0000 UTC m=+811.694325837" watchObservedRunningTime="2026-10-30 05:45:05.121456361 +0000 UTC m=+811.697143415" Oct 30 05:45:05.177809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.177776 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:05.225876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.225826 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" podStartSLOduration=1.742622283 podStartE2EDuration="5.225809138s" totalImagesPullingTime="3.483186855s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:05.224716828 +0000 UTC m=+811.800403883" watchObservedRunningTime="2026-10-30 05:45:05.225809138 +0000 UTC m=+811.801496192" Oct 30 05:45:05.355511 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.354261 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" podStartSLOduration=2.117522037 podStartE2EDuration="5.354241442s" totalImagesPullingTime="3.236719405s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:05.288807192 +0000 UTC m=+811.864494247" watchObservedRunningTime="2026-10-30 05:45:05.354241442 +0000 UTC m=+811.929928497" Oct 30 05:45:05.547329 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.547269 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" podStartSLOduration=2.16454862 podStartE2EDuration="5.547248423s" totalImagesPullingTime="3.382699803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:05.355255568 +0000 UTC m=+811.930942623" watchObservedRunningTime="2026-10-30 05:45:05.547248423 +0000 UTC m=+812.122935478" Oct 30 05:45:05.548130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:05.548104 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d"] Oct 30 05:45:05.548487 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:05.548464 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8aea6cb5_f76d_49ca_891b_2779f4f3689a.slice/crio-5ad4cd13c7484d37f42579be77a328879d50f181ec41c39ad7f3d27062efa5e8 WatchSource:0}: Error finding container 5ad4cd13c7484d37f42579be77a328879d50f181ec41c39ad7f3d27062efa5e8: Status 404 returned error can't find the container with id 5ad4cd13c7484d37f42579be77a328879d50f181ec41c39ad7f3d27062efa5e8 Oct 30 05:45:06.089248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.089219 5565 generic.go:356] "Generic (PLEG): container finished" podID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerID="4356a4befc023a217c023bfc6da89e3c892b539d5712f45745bb48e1d903e56e" exitCode=60 Oct 30 05:45:06.089391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.089296 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerDied","Data":"4356a4befc023a217c023bfc6da89e3c892b539d5712f45745bb48e1d903e56e"} Oct 30 05:45:06.089617 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.089597 5565 scope.go:122] "RemoveContainer" containerID="4356a4befc023a217c023bfc6da89e3c892b539d5712f45745bb48e1d903e56e" Oct 30 05:45:06.091337 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.091311 5565 generic.go:356] "Generic (PLEG): container finished" podID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerID="5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246" exitCode=60 Oct 30 05:45:06.091337 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.091334 5565 generic.go:356] "Generic (PLEG): container finished" podID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerID="bdfc36e664461ab87dd5ba92f6aba85acccac0de3b2373da4fcddb7cc4e83d93" exitCode=60 Oct 30 05:45:06.091482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.091397 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerDied","Data":"5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246"} Oct 30 05:45:06.091482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.091437 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerDied","Data":"bdfc36e664461ab87dd5ba92f6aba85acccac0de3b2373da4fcddb7cc4e83d93"} Oct 30 05:45:06.091482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.091464 5565 scope.go:122] "RemoveContainer" containerID="5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246" Oct 30 05:45:06.091700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.091657 5565 scope.go:122] "RemoveContainer" containerID="bdfc36e664461ab87dd5ba92f6aba85acccac0de3b2373da4fcddb7cc4e83d93" Oct 30 05:45:06.091955 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:06.091927 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v_odh-ai-gateway-infra(40533c19-4848-415b-b48c-8cc6fdcf431b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" Oct 30 05:45:06.093519 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.093497 5565 generic.go:356] "Generic (PLEG): container finished" podID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerID="825bd93914bf52c91cefa843bf81595d4e4571a2d62a499a8cec8d20cfc91f11" exitCode=60 Oct 30 05:45:06.093633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.093553 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerDied","Data":"825bd93914bf52c91cefa843bf81595d4e4571a2d62a499a8cec8d20cfc91f11"} Oct 30 05:45:06.093801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.093782 5565 scope.go:122] "RemoveContainer" containerID="825bd93914bf52c91cefa843bf81595d4e4571a2d62a499a8cec8d20cfc91f11" Oct 30 05:45:06.097376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.097354 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerID="d089b4fd18201320484d7ae323e67aad2aa0cbc942fc156b5a888b7c486119d1" exitCode=60 Oct 30 05:45:06.097475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.097384 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerDied","Data":"d089b4fd18201320484d7ae323e67aad2aa0cbc942fc156b5a888b7c486119d1"} Oct 30 05:45:06.097624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.097608 5565 scope.go:122] "RemoveContainer" containerID="d089b4fd18201320484d7ae323e67aad2aa0cbc942fc156b5a888b7c486119d1" Oct 30 05:45:06.099147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.099126 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" event={"ID":"8aea6cb5-f76d-49ca-891b-2779f4f3689a","Type":"ContainerStarted","Data":"be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b"} Oct 30 05:45:06.099248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.099157 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" event={"ID":"8aea6cb5-f76d-49ca-891b-2779f4f3689a","Type":"ContainerStarted","Data":"5ad4cd13c7484d37f42579be77a328879d50f181ec41c39ad7f3d27062efa5e8"} Oct 30 05:45:06.100792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.100769 5565 generic.go:356] "Generic (PLEG): container finished" podID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerID="a07aac3ef75573ecb9982552d53f60e77815f9b2cf6f4b4662d12a5d43e2f35b" exitCode=60 Oct 30 05:45:06.100885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.100799 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerDied","Data":"a07aac3ef75573ecb9982552d53f60e77815f9b2cf6f4b4662d12a5d43e2f35b"} Oct 30 05:45:06.100991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.100978 5565 scope.go:122] "RemoveContainer" containerID="a07aac3ef75573ecb9982552d53f60e77815f9b2cf6f4b4662d12a5d43e2f35b" Oct 30 05:45:06.147059 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.147039 5565 scope.go:122] "RemoveContainer" containerID="5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246" Oct 30 05:45:06.147360 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:06.147343 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246\": container with ID starting with 5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246 not found: ID does not exist" containerID="5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246" Oct 30 05:45:06.147421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.147367 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246"} err="failed to get container status \"5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246\": rpc error: code = NotFound desc = could not find container \"5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246\": container with ID starting with 5dd2669c977bb772b90ab263014f7e1f229485b83dae72a3c88c61f82c76d246 not found: ID does not exist" Oct 30 05:45:06.178645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.178627 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:06.331602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.331561 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podStartSLOduration=2.331547322 podStartE2EDuration="2.331547322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:06.330125864 +0000 UTC m=+812.905812917" watchObservedRunningTime="2026-10-30 05:45:06.331547322 +0000 UTC m=+812.907234378" Oct 30 05:45:06.632334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.632276 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n"] Oct 30 05:45:06.636403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.636385 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.672216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.672196 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"default-dockercfg-mgnkm\"" Oct 30 05:45:06.672940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.672917 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"simulator-w2-e7a506-kserve-self-signed-certs\"" Oct 30 05:45:06.673028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.672966 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"openshift-service-ca.crt\"" Oct 30 05:45:06.673028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.673002 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"kube-root-ca.crt\"" Oct 30 05:45:06.675169 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.675148 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n"] Oct 30 05:45:06.782388 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.782347 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.782510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.782394 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvnct\" (UniqueName: \"kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.782510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.782426 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.782628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.782583 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.782667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.782652 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.782730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.782714 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.883887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.883814 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.883887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.883864 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.883887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.883883 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvnct\" (UniqueName: \"kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.884123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.883915 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.884123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.883964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.884123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.884048 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.884297 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.884269 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.884348 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.884311 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.884386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.884359 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.886116 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.886094 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.886508 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.886491 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.909496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.909464 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvnct\" (UniqueName: \"kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct\") pod \"simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:06.950005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:06.949977 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:07.107307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.107280 5565 generic.go:356] "Generic (PLEG): container finished" podID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerID="ae2ca07b6d45a5a763dd7517c2cfa70e9808af3092549c7c367d66ea5b07caea" exitCode=60 Oct 30 05:45:07.107423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.107351 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerDied","Data":"ae2ca07b6d45a5a763dd7517c2cfa70e9808af3092549c7c367d66ea5b07caea"} Oct 30 05:45:07.107423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.107408 5565 scope.go:122] "RemoveContainer" containerID="4356a4befc023a217c023bfc6da89e3c892b539d5712f45745bb48e1d903e56e" Oct 30 05:45:07.107699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.107685 5565 scope.go:122] "RemoveContainer" containerID="ae2ca07b6d45a5a763dd7517c2cfa70e9808af3092549c7c367d66ea5b07caea" Oct 30 05:45:07.107933 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:07.107917 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-x4vwr_odh-ai-gateway-infra(e574cb7e-b0a8-42be-b7d0-08e3ed4079dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" Oct 30 05:45:07.109766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.109748 5565 scope.go:122] "RemoveContainer" containerID="bdfc36e664461ab87dd5ba92f6aba85acccac0de3b2373da4fcddb7cc4e83d93" Oct 30 05:45:07.110009 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:07.109978 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v_odh-ai-gateway-infra(40533c19-4848-415b-b48c-8cc6fdcf431b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" Oct 30 05:45:07.111328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.111307 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerStarted","Data":"8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77"} Oct 30 05:45:07.113113 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.113090 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerID="dee79460fe425518ddef0f1eadabae6c49768770411ecb910a2694582abd9c80" exitCode=60 Oct 30 05:45:07.113211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.113170 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerDied","Data":"dee79460fe425518ddef0f1eadabae6c49768770411ecb910a2694582abd9c80"} Oct 30 05:45:07.113382 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.113364 5565 scope.go:122] "RemoveContainer" containerID="dee79460fe425518ddef0f1eadabae6c49768770411ecb910a2694582abd9c80" Oct 30 05:45:07.113609 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:07.113591 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl_odh-ai-gateway-infra(fe3faf3a-984f-4081-a3e3-e068ef4c6e49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" Oct 30 05:45:07.115057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.115034 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerStarted","Data":"b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f"} Oct 30 05:45:07.259989 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.259955 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n"] Oct 30 05:45:07.270174 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:07.270151 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca87ec64_c33c_4f05_91e0_e59dda440a82.slice/crio-51e4f4f732660947ae0147990f6a48a5890014660cc5ea6dd2c40af1e984dbe4 WatchSource:0}: Error finding container 51e4f4f732660947ae0147990f6a48a5890014660cc5ea6dd2c40af1e984dbe4: Status 404 returned error can't find the container with id 51e4f4f732660947ae0147990f6a48a5890014660cc5ea6dd2c40af1e984dbe4 Oct 30 05:45:07.274412 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:07.274396 5565 scope.go:122] "RemoveContainer" containerID="d089b4fd18201320484d7ae323e67aad2aa0cbc942fc156b5a888b7c486119d1" Oct 30 05:45:08.123318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.123285 5565 generic.go:356] "Generic (PLEG): container finished" podID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerID="b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f" exitCode=60 Oct 30 05:45:08.123818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.123343 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerDied","Data":"b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f"} Oct 30 05:45:08.123818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.123408 5565 scope.go:122] "RemoveContainer" containerID="a07aac3ef75573ecb9982552d53f60e77815f9b2cf6f4b4662d12a5d43e2f35b" Oct 30 05:45:08.123818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.123658 5565 scope.go:122] "RemoveContainer" containerID="b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f" Oct 30 05:45:08.123971 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:08.123954 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv_odh-ai-gateway-infra(f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" Oct 30 05:45:08.125543 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.125505 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" event={"ID":"ca87ec64-c33c-4f05-91e0-e59dda440a82","Type":"ContainerStarted","Data":"ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e"} Oct 30 05:45:08.125641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.125567 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" event={"ID":"ca87ec64-c33c-4f05-91e0-e59dda440a82","Type":"ContainerStarted","Data":"51e4f4f732660947ae0147990f6a48a5890014660cc5ea6dd2c40af1e984dbe4"} Oct 30 05:45:08.127317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.127296 5565 scope.go:122] "RemoveContainer" containerID="ae2ca07b6d45a5a763dd7517c2cfa70e9808af3092549c7c367d66ea5b07caea" Oct 30 05:45:08.127589 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:08.127565 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-x4vwr_odh-ai-gateway-infra(e574cb7e-b0a8-42be-b7d0-08e3ed4079dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" Oct 30 05:45:08.129150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.129129 5565 generic.go:356] "Generic (PLEG): container finished" podID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerID="8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77" exitCode=60 Oct 30 05:45:08.129273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.129208 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerDied","Data":"8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77"} Oct 30 05:45:08.129360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.129344 5565 scope.go:122] "RemoveContainer" containerID="8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77" Oct 30 05:45:08.129583 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:08.129563 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w_odh-ai-gateway-infra(91bcd3fb-329b-4b09-bbea-3ff20a08e742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" Oct 30 05:45:08.130969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.130951 5565 scope.go:122] "RemoveContainer" containerID="dee79460fe425518ddef0f1eadabae6c49768770411ecb910a2694582abd9c80" Oct 30 05:45:08.131166 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:08.131148 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl_odh-ai-gateway-infra(fe3faf3a-984f-4081-a3e3-e068ef4c6e49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" Oct 30 05:45:08.138986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.137390 5565 scope.go:122] "RemoveContainer" containerID="825bd93914bf52c91cefa843bf81595d4e4571a2d62a499a8cec8d20cfc91f11" Oct 30 05:45:08.393208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.393109 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podStartSLOduration=2.393095731 podStartE2EDuration="2.393095731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:08.39140563 +0000 UTC m=+814.967092683" watchObservedRunningTime="2026-10-30 05:45:08.393095731 +0000 UTC m=+814.968782785" Oct 30 05:45:08.951000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:08.950966 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:09.138001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:09.137974 5565 scope.go:122] "RemoveContainer" containerID="b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f" Oct 30 05:45:09.138436 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:09.138217 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv_odh-ai-gateway-infra(f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" Oct 30 05:45:09.139618 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:09.139600 5565 scope.go:122] "RemoveContainer" containerID="8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77" Oct 30 05:45:09.139774 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:09.139756 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w_odh-ai-gateway-infra(91bcd3fb-329b-4b09-bbea-3ff20a08e742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" Oct 30 05:45:15.178397 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:15.178332 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:15.190987 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:15.190959 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:16.189389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:16.189362 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:45:16.951003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:16.950974 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:16.963311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:16.963289 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:17.204935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:17.204862 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:45:17.530833 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:17.530799 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:45:18.560320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:18.560291 5565 scope.go:122] "RemoveContainer" containerID="bdfc36e664461ab87dd5ba92f6aba85acccac0de3b2373da4fcddb7cc4e83d93" Oct 30 05:45:18.560752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:18.560470 5565 scope.go:122] "RemoveContainer" containerID="ae2ca07b6d45a5a763dd7517c2cfa70e9808af3092549c7c367d66ea5b07caea" Oct 30 05:45:18.560752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:18.560578 5565 scope.go:122] "RemoveContainer" containerID="b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f" Oct 30 05:45:18.917379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:18.917350 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:45:19.207808 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:19.207710 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerStarted","Data":"667f7a0cd55bad5f3fa27ea1fa1cf1bfa1bb7cc5a3929da6d61db7bd35a8f11b"} Oct 30 05:45:19.210465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:19.210434 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerStarted","Data":"3bc6ff2fa396a861f409bc0b6ba1ae5a7c8991f53a666b95202253d61155632d"} Oct 30 05:45:19.213446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:19.213423 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerStarted","Data":"8a60c82feb549ac04c1d12a00d46e66a6106b45a07006ae8c6d643d9f2f76169"} Oct 30 05:45:19.334416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:19.334366 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" podStartSLOduration=16.248134096 podStartE2EDuration="19.334352742s" totalImagesPullingTime="3.086218646s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:19.333685892 +0000 UTC m=+825.909372947" watchObservedRunningTime="2026-10-30 05:45:19.334352742 +0000 UTC m=+825.910039797" Oct 30 05:45:20.220603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.220573 5565 generic.go:356] "Generic (PLEG): container finished" podID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerID="3bc6ff2fa396a861f409bc0b6ba1ae5a7c8991f53a666b95202253d61155632d" exitCode=60 Oct 30 05:45:20.221174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.220648 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerDied","Data":"3bc6ff2fa396a861f409bc0b6ba1ae5a7c8991f53a666b95202253d61155632d"} Oct 30 05:45:20.221174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.220707 5565 scope.go:122] "RemoveContainer" containerID="ae2ca07b6d45a5a763dd7517c2cfa70e9808af3092549c7c367d66ea5b07caea" Oct 30 05:45:20.221174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.221053 5565 scope.go:122] "RemoveContainer" containerID="3bc6ff2fa396a861f409bc0b6ba1ae5a7c8991f53a666b95202253d61155632d" Oct 30 05:45:20.221375 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:20.221333 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29888985-x4vwr_odh-ai-gateway-infra(e574cb7e-b0a8-42be-b7d0-08e3ed4079dc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" Oct 30 05:45:20.222877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.222856 5565 generic.go:356] "Generic (PLEG): container finished" podID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerID="8a60c82feb549ac04c1d12a00d46e66a6106b45a07006ae8c6d643d9f2f76169" exitCode=60 Oct 30 05:45:20.222968 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.222877 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerDied","Data":"8a60c82feb549ac04c1d12a00d46e66a6106b45a07006ae8c6d643d9f2f76169"} Oct 30 05:45:20.223206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.223188 5565 scope.go:122] "RemoveContainer" containerID="8a60c82feb549ac04c1d12a00d46e66a6106b45a07006ae8c6d643d9f2f76169" Oct 30 05:45:20.223419 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:20.223397 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v_odh-ai-gateway-infra(40533c19-4848-415b-b48c-8cc6fdcf431b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" Oct 30 05:45:20.224895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.224872 5565 generic.go:356] "Generic (PLEG): container finished" podID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerID="667f7a0cd55bad5f3fa27ea1fa1cf1bfa1bb7cc5a3929da6d61db7bd35a8f11b" exitCode=60 Oct 30 05:45:20.224965 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.224936 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerDied","Data":"667f7a0cd55bad5f3fa27ea1fa1cf1bfa1bb7cc5a3929da6d61db7bd35a8f11b"} Oct 30 05:45:20.225194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.225182 5565 scope.go:122] "RemoveContainer" containerID="667f7a0cd55bad5f3fa27ea1fa1cf1bfa1bb7cc5a3929da6d61db7bd35a8f11b" Oct 30 05:45:20.225376 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:20.225358 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv_odh-ai-gateway-infra(f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" Oct 30 05:45:20.234451 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.234438 5565 scope.go:122] "RemoveContainer" containerID="bdfc36e664461ab87dd5ba92f6aba85acccac0de3b2373da4fcddb7cc4e83d93" Oct 30 05:45:20.246587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.246562 5565 scope.go:122] "RemoveContainer" containerID="b13fb94770e399cce1614b61166800f8b027e3278b93dc8069a19e49b236d04f" Oct 30 05:45:20.293995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.293974 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv"] Oct 30 05:45:20.347362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.347340 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v"] Oct 30 05:45:20.412912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.412886 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr"] Oct 30 05:45:20.560680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.560658 5565 scope.go:122] "RemoveContainer" containerID="dee79460fe425518ddef0f1eadabae6c49768770411ecb910a2694582abd9c80" Oct 30 05:45:20.560817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.560803 5565 scope.go:122] "RemoveContainer" containerID="8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77" Oct 30 05:45:20.827660 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.827596 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd"] Oct 30 05:45:20.835394 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.835364 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:20.861771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.861682 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd"] Oct 30 05:45:20.882004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.881982 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"default-dockercfg-bv9bw\"" Oct 30 05:45:20.882004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.881993 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"openshift-service-ca.crt\"" Oct 30 05:45:20.882180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.882035 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"kube-root-ca.crt\"" Oct 30 05:45:20.882223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.882208 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"premium-w4-698ab2-kserve-self-signed-certs\"" Oct 30 05:45:20.936130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.936110 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:20.936221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.936163 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:20.936282 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.936228 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:20.936321 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.936305 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:20.936391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.936377 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxbk2\" (UniqueName: \"kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:20.936474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:20.936463 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.037659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.037623 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.037861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.037671 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.037861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.037758 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxbk2\" (UniqueName: \"kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.037861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.037836 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.038026 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.037882 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.038026 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.037968 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.038151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.038127 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.038211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.038185 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.038321 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.038302 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.040256 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.040219 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.040358 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.040269 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.063206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.063176 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxbk2\" (UniqueName: \"kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2\") pod \"premium-w4-698ab2-kserve-7598ff888d-44gwd\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.149049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.148995 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:21.238418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.237880 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerStarted","Data":"38f7472e3b484496be6b5d5ae1a78c03f1f53336eea55f8cbe58edab750d930e"} Oct 30 05:45:21.241852 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.241823 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerStarted","Data":"46940913f128866b253912e6349e2ec6fc2e911dee57c49bcf23e2956556680f"} Oct 30 05:45:21.464120 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:21.464091 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7759d770_f847_42d7_a258_20dedae07d2a.slice/crio-fe45b0dc835755b355a236291076d2f903b389d9bd9b8def81796a54c98a0ed0 WatchSource:0}: Error finding container fe45b0dc835755b355a236291076d2f903b389d9bd9b8def81796a54c98a0ed0: Status 404 returned error can't find the container with id fe45b0dc835755b355a236291076d2f903b389d9bd9b8def81796a54c98a0ed0 Oct 30 05:45:21.465843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.465817 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd"] Oct 30 05:45:21.468007 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.467967 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:21.471879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.471861 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:21.475721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.475702 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:21.562158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.562118 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca\") pod \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " Oct 30 05:45:21.562505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.562444 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca\") pod \"40533c19-4848-415b-b48c-8cc6fdcf431b\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " Oct 30 05:45:21.562994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.562884 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca" pod "e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" (UID: "e574cb7e-b0a8-42be-b7d0-08e3ed4079dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:21.562994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.562941 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca" pod "40533c19-4848-415b-b48c-8cc6fdcf431b" (UID: "40533c19-4848-415b-b48c-8cc6fdcf431b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:21.563599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.563004 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9\" (UniqueName: \"kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9\") pod \"40533c19-4848-415b-b48c-8cc6fdcf431b\" (UID: \"40533c19-4848-415b-b48c-8cc6fdcf431b\") " Oct 30 05:45:21.563599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.563090 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq\" (UniqueName: \"kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq\") pod \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\" (UID: \"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc\") " Oct 30 05:45:21.563599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.563155 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca\") pod \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " Oct 30 05:45:21.563599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.563198 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh\" (UniqueName: \"kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh\") pod \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\" (UID: \"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2\") " Oct 30 05:45:21.563851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.563759 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca" pod "f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" (UID: "f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:21.565730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.565708 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40533c19-4848-415b-b48c-8cc6fdcf431b-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:21.565892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.565877 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:21.566003 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.565983 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:21.567365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.567318 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq" pod "e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" (UID: "e574cb7e-b0a8-42be-b7d0-08e3ed4079dc"). InnerVolumeSpecName "kube-api-access-bthkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:21.567643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.567615 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9" pod "40533c19-4848-415b-b48c-8cc6fdcf431b" (UID: "40533c19-4848-415b-b48c-8cc6fdcf431b"). InnerVolumeSpecName "kube-api-access-5nmh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:21.567735 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.567690 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh" pod "f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" (UID: "f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2"). InnerVolumeSpecName "kube-api-access-xd2bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:21.667514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.667452 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nmh9\" (UniqueName: \"kubernetes.io/projected/40533c19-4848-415b-b48c-8cc6fdcf431b-kube-api-access-5nmh9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:21.667514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.667478 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bthkq\" (UniqueName: \"kubernetes.io/projected/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc-kube-api-access-bthkq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:21.667514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:21.667494 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd2bh\" (UniqueName: \"kubernetes.io/projected/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2-kube-api-access-xd2bh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:22.253316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.253282 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" event={"ID":"7759d770-f847-42d7-a258-20dedae07d2a","Type":"ContainerStarted","Data":"cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9"} Oct 30 05:45:22.253804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.253337 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" event={"ID":"7759d770-f847-42d7-a258-20dedae07d2a","Type":"ContainerStarted","Data":"fe45b0dc835755b355a236291076d2f903b389d9bd9b8def81796a54c98a0ed0"} Oct 30 05:45:22.254585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.254568 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" Oct 30 05:45:22.254690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.254565 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr" event={"ID":"e574cb7e-b0a8-42be-b7d0-08e3ed4079dc","Type":"ContainerDied","Data":"6bc70983701206a98664e136bafd721b082095b422f837bef2d499d0c89b0276"} Oct 30 05:45:22.254744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.254695 5565 scope.go:122] "RemoveContainer" containerID="3bc6ff2fa396a861f409bc0b6ba1ae5a7c8991f53a666b95202253d61155632d" Oct 30 05:45:22.255962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.255939 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" event={"ID":"40533c19-4848-415b-b48c-8cc6fdcf431b","Type":"ContainerDied","Data":"210d13384a84f5fb0a130a74039ea735cc68fd3215b8a9416d6e96035fa66ba6"} Oct 30 05:45:22.256047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.255949 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v" Oct 30 05:45:22.258031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.258003 5565 generic.go:356] "Generic (PLEG): container finished" podID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerID="38f7472e3b484496be6b5d5ae1a78c03f1f53336eea55f8cbe58edab750d930e" exitCode=60 Oct 30 05:45:22.258115 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.258049 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerDied","Data":"38f7472e3b484496be6b5d5ae1a78c03f1f53336eea55f8cbe58edab750d930e"} Oct 30 05:45:22.258367 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.258350 5565 scope.go:122] "RemoveContainer" containerID="38f7472e3b484496be6b5d5ae1a78c03f1f53336eea55f8cbe58edab750d930e" Oct 30 05:45:22.258611 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:22.258560 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w_odh-ai-gateway-infra(91bcd3fb-329b-4b09-bbea-3ff20a08e742)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" Oct 30 05:45:22.260138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.260118 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerID="46940913f128866b253912e6349e2ec6fc2e911dee57c49bcf23e2956556680f" exitCode=60 Oct 30 05:45:22.260219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.260184 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerDied","Data":"46940913f128866b253912e6349e2ec6fc2e911dee57c49bcf23e2956556680f"} Oct 30 05:45:22.260472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.260455 5565 scope.go:122] "RemoveContainer" containerID="46940913f128866b253912e6349e2ec6fc2e911dee57c49bcf23e2956556680f" Oct 30 05:45:22.260698 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:22.260673 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl_odh-ai-gateway-infra(fe3faf3a-984f-4081-a3e3-e068ef4c6e49)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" Oct 30 05:45:22.261486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.261470 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" Oct 30 05:45:22.261486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.261478 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv" event={"ID":"f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2","Type":"ContainerDied","Data":"3bb56d4883a66257157c89754d0d628f9bacf5db0a86a65ce9bd25002dd94876"} Oct 30 05:45:22.264591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.264575 5565 scope.go:122] "RemoveContainer" containerID="8a60c82feb549ac04c1d12a00d46e66a6106b45a07006ae8c6d643d9f2f76169" Oct 30 05:45:22.272146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.272133 5565 scope.go:122] "RemoveContainer" containerID="8f095da2630064d87bc4761370ede64b9570bfa8e6b5e9d354df7152ebffba77" Oct 30 05:45:22.283568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.283552 5565 scope.go:122] "RemoveContainer" containerID="dee79460fe425518ddef0f1eadabae6c49768770411ecb910a2694582abd9c80" Oct 30 05:45:22.295302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.295286 5565 scope.go:122] "RemoveContainer" containerID="667f7a0cd55bad5f3fa27ea1fa1cf1bfa1bb7cc5a3929da6d61db7bd35a8f11b" Oct 30 05:45:22.310455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.310417 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podStartSLOduration=2.310405614 podStartE2EDuration="2.310405614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:22.30900675 +0000 UTC m=+828.884693804" watchObservedRunningTime="2026-10-30 05:45:22.310405614 +0000 UTC m=+828.886092668" Oct 30 05:45:22.369372 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.369350 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr"] Oct 30 05:45:22.382978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.382959 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29888985-x4vwr"] Oct 30 05:45:22.413265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.413241 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w"] Oct 30 05:45:22.423129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.423109 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v"] Oct 30 05:45:22.430750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.430717 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-be78f2-29888985-9694v"] Oct 30 05:45:22.467451 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.467421 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl"] Oct 30 05:45:22.598270 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.598196 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv"] Oct 30 05:45:22.600610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:22.600584 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29888985-7zmlv"] Oct 30 05:45:23.149098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.149065 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:23.418851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.418827 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:23.444911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.444888 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:23.481877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.481851 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb\" (UniqueName: \"kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb\") pod \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " Oct 30 05:45:23.481992 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.481897 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca\") pod \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\" (UID: \"fe3faf3a-984f-4081-a3e3-e068ef4c6e49\") " Oct 30 05:45:23.482278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.482258 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca" pod "fe3faf3a-984f-4081-a3e3-e068ef4c6e49" (UID: "fe3faf3a-984f-4081-a3e3-e068ef4c6e49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:23.482467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.482447 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:23.483861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.483839 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb" pod "fe3faf3a-984f-4081-a3e3-e068ef4c6e49" (UID: "fe3faf3a-984f-4081-a3e3-e068ef4c6e49"). InnerVolumeSpecName "kube-api-access-q5qtb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:23.567722 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.567691 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" path="/var/lib/kubelet/pods/40533c19-4848-415b-b48c-8cc6fdcf431b/volumes" Oct 30 05:45:23.568133 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.568118 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" path="/var/lib/kubelet/pods/e574cb7e-b0a8-42be-b7d0-08e3ed4079dc/volumes" Oct 30 05:45:23.568566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.568550 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" path="/var/lib/kubelet/pods/f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2/volumes" Oct 30 05:45:23.583066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.583034 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca\") pod \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " Oct 30 05:45:23.583178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.583085 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r\" (UniqueName: \"kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r\") pod \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\" (UID: \"91bcd3fb-329b-4b09-bbea-3ff20a08e742\") " Oct 30 05:45:23.583355 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.583335 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca" pod "91bcd3fb-329b-4b09-bbea-3ff20a08e742" (UID: "91bcd3fb-329b-4b09-bbea-3ff20a08e742"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:23.583432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.583420 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5qtb\" (UniqueName: \"kubernetes.io/projected/fe3faf3a-984f-4081-a3e3-e068ef4c6e49-kube-api-access-q5qtb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:23.583488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.583432 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91bcd3fb-329b-4b09-bbea-3ff20a08e742-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:23.585356 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.585335 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r" pod "91bcd3fb-329b-4b09-bbea-3ff20a08e742" (UID: "91bcd3fb-329b-4b09-bbea-3ff20a08e742"). InnerVolumeSpecName "kube-api-access-m862r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:23.684379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:23.684295 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m862r\" (UniqueName: \"kubernetes.io/projected/91bcd3fb-329b-4b09-bbea-3ff20a08e742-kube-api-access-m862r\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:24.287105 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.287077 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" Oct 30 05:45:24.287289 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.287106 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w" event={"ID":"91bcd3fb-329b-4b09-bbea-3ff20a08e742","Type":"ContainerDied","Data":"ee9f7659a2fdb965697cabfb09fe8fffba321bf17096094ca78850a4ff7cb35a"} Oct 30 05:45:24.287289 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.287181 5565 scope.go:122] "RemoveContainer" containerID="38f7472e3b484496be6b5d5ae1a78c03f1f53336eea55f8cbe58edab750d930e" Oct 30 05:45:24.288619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.288595 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" Oct 30 05:45:24.288744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.288621 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl" event={"ID":"fe3faf3a-984f-4081-a3e3-e068ef4c6e49","Type":"ContainerDied","Data":"59a6ba738562a62b6cb05c77f6a35e0be178995e2429965696bc0499e3d08357"} Oct 30 05:45:24.296124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.296106 5565 scope.go:122] "RemoveContainer" containerID="46940913f128866b253912e6349e2ec6fc2e911dee57c49bcf23e2956556680f" Oct 30 05:45:24.350636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.350615 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w"] Oct 30 05:45:24.364350 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.364325 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-fffaa8-29888985-28w6w"] Oct 30 05:45:24.393152 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.393135 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl"] Oct 30 05:45:24.397822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:24.397803 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29888985-lhkpl"] Oct 30 05:45:25.567049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:25.567019 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" path="/var/lib/kubelet/pods/91bcd3fb-329b-4b09-bbea-3ff20a08e742/volumes" Oct 30 05:45:25.567425 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:25.567388 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" path="/var/lib/kubelet/pods/fe3faf3a-984f-4081-a3e3-e068ef4c6e49/volumes" Oct 30 05:45:31.149336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.149305 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:31.161455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.161432 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:31.349146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.349119 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:45:31.384287 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384258 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2"] Oct 30 05:45:31.384778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384761 5565 cpu_manager.go:414] "containerMap: removing container" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384779 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384802 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384810 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384830 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384838 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384850 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384858 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384870 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.384876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384878 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384887 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384894 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384911 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384918 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384937 5565 cpu_manager.go:414] "containerMap: removing container" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384944 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384959 5565 cpu_manager.go:414] "containerMap: removing container" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384966 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384982 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.384989 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385009 5565 cpu_manager.go:414] "containerMap: removing container" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385016 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385124 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385137 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385145 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385157 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385165 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385176 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385184 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385193 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385203 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385215 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385225 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385234 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.385308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.385244 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:31.389479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.389459 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.414664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.414615 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-1d8c2f-openshift-default-dockercfg-7kqf6\"" Oct 30 05:45:31.425248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.425220 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2"] Oct 30 05:45:31.486696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.486662 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.486696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.486697 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.486876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.486722 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.486876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.486863 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.486994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.486980 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qftj\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.487032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.487005 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.487032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.487023 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.487149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.487133 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.487189 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.487179 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.587952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.587916 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.587952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.587957 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.587977 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588012 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588056 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qftj\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588087 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588113 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588170 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588215 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588396 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588450 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588555 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588591 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.588826 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.588804 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.590347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.590320 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.590449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.590394 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.616316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.616295 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.616443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.616420 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qftj\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:31.701645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:31.701586 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:32.016093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.016066 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2"] Oct 30 05:45:32.017542 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:32.017501 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod140e5755_d9a3_428f_8bd2_616cabc92c2a.slice/crio-6ca4dbf5d6b2a2b87b880c1d09bea4ff0e0f33a691666130e1c1be3d3a48ba7d WatchSource:0}: Error finding container 6ca4dbf5d6b2a2b87b880c1d09bea4ff0e0f33a691666130e1c1be3d3a48ba7d: Status 404 returned error can't find the container with id 6ca4dbf5d6b2a2b87b880c1d09bea4ff0e0f33a691666130e1c1be3d3a48ba7d Oct 30 05:45:32.019804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.019775 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:45:32.019882 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.019844 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:45:32.019882 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.019874 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:45:32.343651 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.343558 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" event={"ID":"140e5755-d9a3-428f-8bd2-616cabc92c2a","Type":"ContainerStarted","Data":"b2aee50c4fe9897573e4b077c60c0cb444f03dd8d02653c56598785cdac01a92"} Oct 30 05:45:32.343651 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.343621 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" event={"ID":"140e5755-d9a3-428f-8bd2-616cabc92c2a","Type":"ContainerStarted","Data":"6ca4dbf5d6b2a2b87b880c1d09bea4ff0e0f33a691666130e1c1be3d3a48ba7d"} Oct 30 05:45:32.395998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.395936 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podStartSLOduration=1.395917763 podStartE2EDuration="1.395917763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:32.395461669 +0000 UTC m=+838.971148726" watchObservedRunningTime="2026-10-30 05:45:32.395917763 +0000 UTC m=+838.971604820" Oct 30 05:45:32.702276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:32.702201 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:33.703814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:33.703791 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:45:33.704094 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:33.703838 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:45:34.018794 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:34.018357 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:45:34.702192 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:34.702152 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:45:34.702360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:34.702211 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:45:35.702938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:35.702906 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:45:35.703337 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:35.702956 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:45:36.702344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:36.702308 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:45:36.702519 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:36.702363 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:45:37.702334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:37.702302 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:45:37.702731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:37.702349 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:45:38.702859 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:38.702823 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" start-of-body= Oct 30 05:45:38.703370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:38.702876 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.54:15021/healthz/ready\": dial tcp 10.134.0.54:15021: connect: connection refused" Oct 30 05:45:39.706430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:39.706403 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:39.706839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:39.706632 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:39.707310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:39.707292 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:45:43.954439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.954399 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p"] Oct 30 05:45:43.956196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956166 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:43.956384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956367 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3e78dd0-9242-48b7-b4ca-9c43a97bc4a2" containerName="cleanup" Oct 30 05:45:43.956544 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956515 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:43.956648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956635 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe3faf3a-984f-4081-a3e3-e068ef4c6e49" containerName="cleanup" Oct 30 05:45:43.956783 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956769 5565 cpu_manager.go:414] "containerMap: removing container" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:43.956867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956855 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40533c19-4848-415b-b48c-8cc6fdcf431b" containerName="cleanup" Oct 30 05:45:43.956990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.956979 5565 cpu_manager.go:414] "containerMap: removing container" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:43.957374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.957357 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:43.957764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.957743 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="91bcd3fb-329b-4b09-bbea-3ff20a08e742" containerName="cleanup" Oct 30 05:45:43.957914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.957901 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e574cb7e-b0a8-42be-b7d0-08e3ed4079dc" containerName="cleanup" Oct 30 05:45:43.966316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.966295 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:43.981988 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.981967 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p"] Oct 30 05:45:43.993954 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.993932 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"openshift-service-ca.crt\"" Oct 30 05:45:43.994082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.994062 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"kube-root-ca.crt\"" Oct 30 05:45:43.998566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.998550 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"default-dockercfg-279cs\"" Oct 30 05:45:43.998703 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:43.998687 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"premium-w3-fffaa8-kserve-self-signed-certs\"" Oct 30 05:45:44.040233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.040213 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.040340 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.040259 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.040340 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.040282 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.040474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.040454 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.040523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.040491 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.040523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.040510 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm5df\" (UniqueName: \"kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.141742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.141708 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.141885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.141766 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.141885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.141792 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm5df\" (UniqueName: \"kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.141885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.141849 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.142071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.141890 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.142071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.141926 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.142376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.142351 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.142643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.142618 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.143155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.143134 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.145133 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.145087 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.145380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.145360 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.168384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.168343 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm5df\" (UniqueName: \"kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df\") pod \"premium-w3-fffaa8-kserve-c7b746df9-6654p\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.278790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.278761 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:44.583251 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:44.583227 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p"] Oct 30 05:45:44.585064 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:44.585038 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbe07924b_4a72_4f8f_8ff0_e4e71854b388.slice/crio-0b16a6944dabcd421d73b18f265a60dd28cf37306e5b7d7fa69c340e8df41866 WatchSource:0}: Error finding container 0b16a6944dabcd421d73b18f265a60dd28cf37306e5b7d7fa69c340e8df41866: Status 404 returned error can't find the container with id 0b16a6944dabcd421d73b18f265a60dd28cf37306e5b7d7fa69c340e8df41866 Oct 30 05:45:45.438707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:45.438663 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" event={"ID":"be07924b-4a72-4f8f-8ff0-e4e71854b388","Type":"ContainerStarted","Data":"9e17c784f73ada8f933c48ce9e50b7a8c9bd8761d701982483f74419d8afae98"} Oct 30 05:45:45.438707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:45.438713 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" event={"ID":"be07924b-4a72-4f8f-8ff0-e4e71854b388","Type":"ContainerStarted","Data":"0b16a6944dabcd421d73b18f265a60dd28cf37306e5b7d7fa69c340e8df41866"} Oct 30 05:45:45.491782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:45.491720 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podStartSLOduration=2.491703038 podStartE2EDuration="2.491703038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:45.489803484 +0000 UTC m=+852.065490539" watchObservedRunningTime="2026-10-30 05:45:45.491703038 +0000 UTC m=+852.067390091" Oct 30 05:45:46.279272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:46.279226 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:47.539392 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:45:47.539351 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:45:54.279711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.279677 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:54.292777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.292755 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:54.300386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.300356 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh"] Oct 30 05:45:54.304541 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.304511 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.321917 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.321894 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:45:54.324747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.324727 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:45:54.326102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.326065 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:45:54.326208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.326120 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:45:54.327744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.327722 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh"] Oct 30 05:45:54.453343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.453314 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzmtd\" (UniqueName: \"kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.453500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.453356 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.453572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.453514 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.499876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.499843 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:45:54.554292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.554216 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.554439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.554306 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.554439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.554377 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzmtd\" (UniqueName: \"kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.555011 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.554985 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.556941 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.556920 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.580349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.580322 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzmtd\" (UniqueName: \"kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd\") pod \"maas-api-revoke-sub-e870778b-6s6vh\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.618622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.618599 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:54.921440 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:54.921416 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh"] Oct 30 05:45:54.923747 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:54.923719 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode849a6b2_593e_43b6_a52a_f4e58b6b2126.slice/crio-eaca48e2021734050666038480d4e39ce8b620432830b62f3f23b98e944b4711 WatchSource:0}: Error finding container eaca48e2021734050666038480d4e39ce8b620432830b62f3f23b98e944b4711: Status 404 returned error can't find the container with id eaca48e2021734050666038480d4e39ce8b620432830b62f3f23b98e944b4711 Oct 30 05:45:55.497109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:55.497068 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" event={"ID":"e849a6b2-593e-43b6-a52a-f4e58b6b2126","Type":"ContainerStarted","Data":"70a6e4707e8ef50c114433011f8d8fdac108488c77d0d5dd0123aaa417e5b92d"} Oct 30 05:45:55.497567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:55.497124 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" event={"ID":"e849a6b2-593e-43b6-a52a-f4e58b6b2126","Type":"ContainerStarted","Data":"eaca48e2021734050666038480d4e39ce8b620432830b62f3f23b98e944b4711"} Oct 30 05:45:55.549325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:55.549213 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" podStartSLOduration=1.5492006919999999 podStartE2EDuration="1.549200692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:55.544991576 +0000 UTC m=+862.120678629" watchObservedRunningTime="2026-10-30 05:45:55.549200692 +0000 UTC m=+862.124887745" Oct 30 05:45:56.216595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.216476 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq"] Oct 30 05:45:56.221229 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.221198 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.242518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.242484 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq"] Oct 30 05:45:56.254209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.254180 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-e7a506\"/\"premium-w2-e7a506-kserve-self-signed-certs\"" Oct 30 05:45:56.374947 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.374909 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.375137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.374971 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.375137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.375100 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.375221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.375174 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4lkq\" (UniqueName: \"kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.375265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.375240 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.375300 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.375284 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476229 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476343 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4lkq\" (UniqueName: \"kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476370 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476393 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476459 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476489 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476776 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476818 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.476891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.476882 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.478907 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.478880 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.479097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.479076 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.510588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.510547 5565 generic.go:356] "Generic (PLEG): container finished" podID="e849a6b2-593e-43b6-a52a-f4e58b6b2126" containerID="70a6e4707e8ef50c114433011f8d8fdac108488c77d0d5dd0123aaa417e5b92d" exitCode=0 Oct 30 05:45:56.511070 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.510595 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" event={"ID":"e849a6b2-593e-43b6-a52a-f4e58b6b2126","Type":"ContainerDied","Data":"70a6e4707e8ef50c114433011f8d8fdac108488c77d0d5dd0123aaa417e5b92d"} Oct 30 05:45:56.520151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.520119 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4lkq\" (UniqueName: \"kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq\") pod \"premium-w2-e7a506-kserve-7d779bf4f6-65vwq\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.535565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.535512 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:56.844051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:56.844016 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq"] Oct 30 05:45:56.847461 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:45:56.847422 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf62bec26_ed63_43bf_b0f3_6b73a2464339.slice/crio-82a5bc8f4f53a5cfc160372ed589dc81d4e3360835a69fd72ccc7bce9a344810 WatchSource:0}: Error finding container 82a5bc8f4f53a5cfc160372ed589dc81d4e3360835a69fd72ccc7bce9a344810: Status 404 returned error can't find the container with id 82a5bc8f4f53a5cfc160372ed589dc81d4e3360835a69fd72ccc7bce9a344810 Oct 30 05:45:57.518058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.518014 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" event={"ID":"f62bec26-ed63-43bf-b0f3-6b73a2464339","Type":"ContainerStarted","Data":"514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55"} Oct 30 05:45:57.518058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.518068 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" event={"ID":"f62bec26-ed63-43bf-b0f3-6b73a2464339","Type":"ContainerStarted","Data":"82a5bc8f4f53a5cfc160372ed589dc81d4e3360835a69fd72ccc7bce9a344810"} Oct 30 05:45:57.536431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.536354 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:45:57.570986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.570933 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podStartSLOduration=1.5709179739999999 podStartE2EDuration="1.570917974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:45:57.569627722 +0000 UTC m=+864.145314777" watchObservedRunningTime="2026-10-30 05:45:57.570917974 +0000 UTC m=+864.146605028" Oct 30 05:45:57.682180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.682114 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:45:57.690665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.690626 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca\") pod \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " Oct 30 05:45:57.690872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.690698 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca\") pod \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " Oct 30 05:45:57.690937 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.690873 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd\" (UniqueName: \"kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd\") pod \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\" (UID: \"e849a6b2-593e-43b6-a52a-f4e58b6b2126\") " Oct 30 05:45:57.691086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.691052 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca" pod "e849a6b2-593e-43b6-a52a-f4e58b6b2126" (UID: "e849a6b2-593e-43b6-a52a-f4e58b6b2126"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:45:57.691369 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.691347 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:57.693218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.693184 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd" pod "e849a6b2-593e-43b6-a52a-f4e58b6b2126" (UID: "e849a6b2-593e-43b6-a52a-f4e58b6b2126"). InnerVolumeSpecName "kube-api-access-xzmtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:45:57.693300 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.693229 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca" pod "e849a6b2-593e-43b6-a52a-f4e58b6b2126" (UID: "e849a6b2-593e-43b6-a52a-f4e58b6b2126"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:45:57.698555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.698488 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:45:57.792432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.792332 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzmtd\" (UniqueName: \"kubernetes.io/projected/e849a6b2-593e-43b6-a52a-f4e58b6b2126-kube-api-access-xzmtd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:57.792432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:57.792367 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e849a6b2-593e-43b6-a52a-f4e58b6b2126-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:45:58.542552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:58.541881 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" event={"ID":"e849a6b2-593e-43b6-a52a-f4e58b6b2126","Type":"ContainerDied","Data":"eaca48e2021734050666038480d4e39ce8b620432830b62f3f23b98e944b4711"} Oct 30 05:45:58.542552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:58.542040 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaca48e2021734050666038480d4e39ce8b620432830b62f3f23b98e944b4711" Oct 30 05:45:58.542552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:45:58.542264 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh" Oct 30 05:46:03.624587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.624545 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s"] Oct 30 05:46:03.625319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.625297 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e849a6b2-593e-43b6-a52a-f4e58b6b2126" containerName="revoke-keys" Oct 30 05:46:03.625405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.625321 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e849a6b2-593e-43b6-a52a-f4e58b6b2126" containerName="revoke-keys" Oct 30 05:46:03.625489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.625475 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e849a6b2-593e-43b6-a52a-f4e58b6b2126" containerName="revoke-keys" Oct 30 05:46:03.629259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.629228 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.651210 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.651071 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:03.651210 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.651104 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s"] Oct 30 05:46:03.661024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.660988 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:03.661024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.661018 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:03.661315 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.661107 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:46:03.693036 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.692996 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.693254 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.693226 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fww58\" (UniqueName: \"kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.693327 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.693284 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.794009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.793964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fww58\" (UniqueName: \"kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.794009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.794017 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.794291 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.794076 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.794890 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.794868 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.796486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.796456 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.821218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.821181 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fww58\" (UniqueName: \"kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58\") pod \"maas-api-revoke-sub-caef318f-z9k8s\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:03.943364 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:03.943262 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:04.260716 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:04.260679 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s"] Oct 30 05:46:04.261055 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:04.261027 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa101b13_4b16_45ad_b411_44d2db791dc3.slice/crio-a0538b8364aa047114dae3531bcc3cc866c9dcd92e9d6d9fded8f75c4d8ffdba WatchSource:0}: Error finding container a0538b8364aa047114dae3531bcc3cc866c9dcd92e9d6d9fded8f75c4d8ffdba: Status 404 returned error can't find the container with id a0538b8364aa047114dae3531bcc3cc866c9dcd92e9d6d9fded8f75c4d8ffdba Oct 30 05:46:04.577774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:04.577676 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" event={"ID":"aa101b13-4b16-45ad-b411-44d2db791dc3","Type":"ContainerStarted","Data":"a0538b8364aa047114dae3531bcc3cc866c9dcd92e9d6d9fded8f75c4d8ffdba"} Oct 30 05:46:05.586468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:05.586433 5565 generic.go:356] "Generic (PLEG): container finished" podID="aa101b13-4b16-45ad-b411-44d2db791dc3" containerID="f4e2af4a4d5b3408b37977cdfb3dfbdd9ebff78a1168974c5e6b30dbea271856" exitCode=0 Oct 30 05:46:05.586992 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:05.586538 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" event={"ID":"aa101b13-4b16-45ad-b411-44d2db791dc3","Type":"ContainerDied","Data":"f4e2af4a4d5b3408b37977cdfb3dfbdd9ebff78a1168974c5e6b30dbea271856"} Oct 30 05:46:06.535821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.535781 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:46:06.551507 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.551460 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:46:06.606590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.606546 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:46:06.756221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.756194 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:06.837624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.837510 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58\" (UniqueName: \"kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58\") pod \"aa101b13-4b16-45ad-b411-44d2db791dc3\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " Oct 30 05:46:06.837809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.837724 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca\") pod \"aa101b13-4b16-45ad-b411-44d2db791dc3\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " Oct 30 05:46:06.837877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.837833 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca\") pod \"aa101b13-4b16-45ad-b411-44d2db791dc3\" (UID: \"aa101b13-4b16-45ad-b411-44d2db791dc3\") " Oct 30 05:46:06.838267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.838235 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca" pod "aa101b13-4b16-45ad-b411-44d2db791dc3" (UID: "aa101b13-4b16-45ad-b411-44d2db791dc3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:06.839834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.839798 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58" pod "aa101b13-4b16-45ad-b411-44d2db791dc3" (UID: "aa101b13-4b16-45ad-b411-44d2db791dc3"). InnerVolumeSpecName "kube-api-access-fww58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:06.839945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.839917 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca" pod "aa101b13-4b16-45ad-b411-44d2db791dc3" (UID: "aa101b13-4b16-45ad-b411-44d2db791dc3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:06.939490 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.939451 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:06.939490 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.939484 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa101b13-4b16-45ad-b411-44d2db791dc3-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:06.939490 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:06.939494 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fww58\" (UniqueName: \"kubernetes.io/projected/aa101b13-4b16-45ad-b411-44d2db791dc3-kube-api-access-fww58\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:07.601801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:07.601762 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" Oct 30 05:46:07.602001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:07.601792 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s" event={"ID":"aa101b13-4b16-45ad-b411-44d2db791dc3","Type":"ContainerDied","Data":"a0538b8364aa047114dae3531bcc3cc866c9dcd92e9d6d9fded8f75c4d8ffdba"} Oct 30 05:46:07.602001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:07.601855 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0538b8364aa047114dae3531bcc3cc866c9dcd92e9d6d9fded8f75c4d8ffdba" Oct 30 05:46:10.212380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.212344 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb"] Oct 30 05:46:10.212938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.212921 5565 cpu_manager.go:414] "containerMap: removing container" podUID="aa101b13-4b16-45ad-b411-44d2db791dc3" containerName="revoke-keys" Oct 30 05:46:10.212982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.212939 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa101b13-4b16-45ad-b411-44d2db791dc3" containerName="revoke-keys" Oct 30 05:46:10.213053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.213044 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa101b13-4b16-45ad-b411-44d2db791dc3" containerName="revoke-keys" Oct 30 05:46:10.217523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.217505 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.245830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.245684 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:10.246071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.246035 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:10.249475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.249448 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb"] Oct 30 05:46:10.283940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.283913 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:10.284062 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.283947 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:46:10.290612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.290591 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8"] Oct 30 05:46:10.294454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.294440 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.319936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.319910 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8"] Oct 30 05:46:10.381650 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.381613 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.381806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.381672 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.381849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.381798 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qcmc\" (UniqueName: \"kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.381915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.381900 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.381967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.381954 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.382005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.381984 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cfjh\" (UniqueName: \"kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.483589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.483469 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.483589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.483520 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cfjh\" (UniqueName: \"kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.483589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.483588 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.483861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.483637 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.483927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.483856 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qcmc\" (UniqueName: \"kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.483987 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.483965 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.484309 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.484273 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.484450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.484324 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.486080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.486055 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.486207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.486192 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.516929 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.516901 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cfjh\" (UniqueName: \"kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh\") pod \"maas-api-revoke-sub-0275c3e1-2kmtb\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.517039 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.516994 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qcmc\" (UniqueName: \"kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc\") pod \"maas-api-revoke-sub-aea364b4-vqwd8\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.533113 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.533087 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:10.606792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.606765 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:10.848465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.848426 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb"] Oct 30 05:46:10.852311 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:10.852285 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6507588a_cdd2_4f2a_9133_b34def476866.slice/crio-a9dca2db85548abfccd54e791276f18082e3c868c6a7ae56c145d4bf347cfed2 WatchSource:0}: Error finding container a9dca2db85548abfccd54e791276f18082e3c868c6a7ae56c145d4bf347cfed2: Status 404 returned error can't find the container with id a9dca2db85548abfccd54e791276f18082e3c868c6a7ae56c145d4bf347cfed2 Oct 30 05:46:10.911435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:10.911411 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8"] Oct 30 05:46:10.912734 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:10.912707 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf331dc96_299e_4ef6_8a81_ed75cca5a097.slice/crio-9e37a624e8e63c384fc587688f15a994b57ec87f03848f8b0c3c5fa8bc9fb410 WatchSource:0}: Error finding container 9e37a624e8e63c384fc587688f15a994b57ec87f03848f8b0c3c5fa8bc9fb410: Status 404 returned error can't find the container with id 9e37a624e8e63c384fc587688f15a994b57ec87f03848f8b0c3c5fa8bc9fb410 Oct 30 05:46:11.626437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:11.626402 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" event={"ID":"f331dc96-299e-4ef6-8a81-ed75cca5a097","Type":"ContainerStarted","Data":"1ad8fa516781fe87a8e83c2b137b6dc41707682433211af76575b01759a626f3"} Oct 30 05:46:11.626877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:11.626459 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" event={"ID":"f331dc96-299e-4ef6-8a81-ed75cca5a097","Type":"ContainerStarted","Data":"9e37a624e8e63c384fc587688f15a994b57ec87f03848f8b0c3c5fa8bc9fb410"} Oct 30 05:46:11.627980 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:11.627955 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" event={"ID":"6507588a-cdd2-4f2a-9133-b34def476866","Type":"ContainerStarted","Data":"58beee501640ff50d7aaee9868dd6e3785765a52b39a56121b06cf0f7244cae7"} Oct 30 05:46:11.628076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:11.627997 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" event={"ID":"6507588a-cdd2-4f2a-9133-b34def476866","Type":"ContainerStarted","Data":"a9dca2db85548abfccd54e791276f18082e3c868c6a7ae56c145d4bf347cfed2"} Oct 30 05:46:11.698598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:11.698546 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" podStartSLOduration=1.6985173649999998 podStartE2EDuration="1.698517365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:11.697007911 +0000 UTC m=+878.272694964" watchObservedRunningTime="2026-10-30 05:46:11.698517365 +0000 UTC m=+878.274204419" Oct 30 05:46:11.771134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:11.767393 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" podStartSLOduration=1.767375387 podStartE2EDuration="1.767375387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:11.764335683 +0000 UTC m=+878.340022737" watchObservedRunningTime="2026-10-30 05:46:11.767375387 +0000 UTC m=+878.343062444" Oct 30 05:46:12.637239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:12.637201 5565 generic.go:356] "Generic (PLEG): container finished" podID="f331dc96-299e-4ef6-8a81-ed75cca5a097" containerID="1ad8fa516781fe87a8e83c2b137b6dc41707682433211af76575b01759a626f3" exitCode=0 Oct 30 05:46:12.637730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:12.637303 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" event={"ID":"f331dc96-299e-4ef6-8a81-ed75cca5a097","Type":"ContainerDied","Data":"1ad8fa516781fe87a8e83c2b137b6dc41707682433211af76575b01759a626f3"} Oct 30 05:46:12.639102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:12.639074 5565 generic.go:356] "Generic (PLEG): container finished" podID="6507588a-cdd2-4f2a-9133-b34def476866" containerID="58beee501640ff50d7aaee9868dd6e3785765a52b39a56121b06cf0f7244cae7" exitCode=0 Oct 30 05:46:12.639222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:12.639149 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" event={"ID":"6507588a-cdd2-4f2a-9133-b34def476866","Type":"ContainerDied","Data":"58beee501640ff50d7aaee9868dd6e3785765a52b39a56121b06cf0f7244cae7"} Oct 30 05:46:14.126563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.126153 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:14.150889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.150793 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:14.209495 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.209427 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca\") pod \"6507588a-cdd2-4f2a-9133-b34def476866\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " Oct 30 05:46:14.209834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.209797 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca\") pod \"f331dc96-299e-4ef6-8a81-ed75cca5a097\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " Oct 30 05:46:14.210102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.210041 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca\") pod \"6507588a-cdd2-4f2a-9133-b34def476866\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " Oct 30 05:46:14.210285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.210252 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc\" (UniqueName: \"kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc\") pod \"f331dc96-299e-4ef6-8a81-ed75cca5a097\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " Oct 30 05:46:14.210546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.210466 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh\" (UniqueName: \"kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh\") pod \"6507588a-cdd2-4f2a-9133-b34def476866\" (UID: \"6507588a-cdd2-4f2a-9133-b34def476866\") " Oct 30 05:46:14.210742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.210670 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca\") pod \"f331dc96-299e-4ef6-8a81-ed75cca5a097\" (UID: \"f331dc96-299e-4ef6-8a81-ed75cca5a097\") " Oct 30 05:46:14.211990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.211964 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca" pod "f331dc96-299e-4ef6-8a81-ed75cca5a097" (UID: "f331dc96-299e-4ef6-8a81-ed75cca5a097"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:14.213778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.213750 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca" pod "6507588a-cdd2-4f2a-9133-b34def476866" (UID: "6507588a-cdd2-4f2a-9133-b34def476866"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:14.344061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.343955 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:14.344338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.344318 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:14.354921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.354858 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca" pod "6507588a-cdd2-4f2a-9133-b34def476866" (UID: "6507588a-cdd2-4f2a-9133-b34def476866"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:14.355408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.355371 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh" pod "6507588a-cdd2-4f2a-9133-b34def476866" (UID: "6507588a-cdd2-4f2a-9133-b34def476866"). InnerVolumeSpecName "kube-api-access-9cfjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:14.356825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.356784 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca" pod "f331dc96-299e-4ef6-8a81-ed75cca5a097" (UID: "f331dc96-299e-4ef6-8a81-ed75cca5a097"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:14.357071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.357042 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc" pod "f331dc96-299e-4ef6-8a81-ed75cca5a097" (UID: "f331dc96-299e-4ef6-8a81-ed75cca5a097"). InnerVolumeSpecName "kube-api-access-4qcmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:14.450559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.445775 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qcmc\" (UniqueName: \"kubernetes.io/projected/f331dc96-299e-4ef6-8a81-ed75cca5a097-kube-api-access-4qcmc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:14.450559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.445815 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cfjh\" (UniqueName: \"kubernetes.io/projected/6507588a-cdd2-4f2a-9133-b34def476866-kube-api-access-9cfjh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:14.450559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.445831 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6507588a-cdd2-4f2a-9133-b34def476866-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:14.450559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.445848 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f331dc96-299e-4ef6-8a81-ed75cca5a097-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:14.653775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.653685 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" Oct 30 05:46:14.653775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.653684 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8" event={"ID":"f331dc96-299e-4ef6-8a81-ed75cca5a097","Type":"ContainerDied","Data":"9e37a624e8e63c384fc587688f15a994b57ec87f03848f8b0c3c5fa8bc9fb410"} Oct 30 05:46:14.654034 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.653810 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e37a624e8e63c384fc587688f15a994b57ec87f03848f8b0c3c5fa8bc9fb410" Oct 30 05:46:14.655246 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.655221 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" Oct 30 05:46:14.655390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.655238 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb" event={"ID":"6507588a-cdd2-4f2a-9133-b34def476866","Type":"ContainerDied","Data":"a9dca2db85548abfccd54e791276f18082e3c868c6a7ae56c145d4bf347cfed2"} Oct 30 05:46:14.655390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:14.655287 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a9dca2db85548abfccd54e791276f18082e3c868c6a7ae56c145d4bf347cfed2" Oct 30 05:46:16.601426 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.601380 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g"] Oct 30 05:46:16.602212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.602186 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f331dc96-299e-4ef6-8a81-ed75cca5a097" containerName="revoke-keys" Oct 30 05:46:16.602212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.602207 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f331dc96-299e-4ef6-8a81-ed75cca5a097" containerName="revoke-keys" Oct 30 05:46:16.602383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.602227 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6507588a-cdd2-4f2a-9133-b34def476866" containerName="revoke-keys" Oct 30 05:46:16.602383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.602235 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6507588a-cdd2-4f2a-9133-b34def476866" containerName="revoke-keys" Oct 30 05:46:16.602383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.602367 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f331dc96-299e-4ef6-8a81-ed75cca5a097" containerName="revoke-keys" Oct 30 05:46:16.602383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.602381 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6507588a-cdd2-4f2a-9133-b34def476866" containerName="revoke-keys" Oct 30 05:46:16.620552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.614646 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.631623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.631589 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g"] Oct 30 05:46:16.647931 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.647901 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-698ab2-openshift-default-dockercfg-fl985\"" Oct 30 05:46:16.674014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.673972 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m99p\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.674413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.674391 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.674680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.674659 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.674827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.674813 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.674996 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.674983 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.675120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.675107 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.675248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.675235 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.675391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.675378 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.675588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.675571 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.780384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.780343 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.781162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.781134 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m99p\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.797551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.781717 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.797867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.797840 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.798413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.798391 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.798958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.798936 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.799454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.799435 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.806654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.806625 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.812462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.799352 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.812462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.800143 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.812462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.807104 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.812462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.797189 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.815662 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.815633 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.816709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.816177 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.831555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.831130 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.856336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.856257 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m99p\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.857053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.857028 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.857383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.857365 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:16.966554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:16.965813 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:17.364194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.364169 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g"] Oct 30 05:46:17.365753 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:17.365726 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27f64629_383b_4efd_95b2_84036908df7c.slice/crio-8cf5920ad3b3a4e44789fb6972fc18d49fb73d6d5307614e365d5926e8de47db WatchSource:0}: Error finding container 8cf5920ad3b3a4e44789fb6972fc18d49fb73d6d5307614e365d5926e8de47db: Status 404 returned error can't find the container with id 8cf5920ad3b3a4e44789fb6972fc18d49fb73d6d5307614e365d5926e8de47db Oct 30 05:46:17.368465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.368431 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:17.368601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.368509 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:17.368601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.368580 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:17.533645 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:46:17.533608 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:46:17.746583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.746463 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" event={"ID":"27f64629-383b-4efd-95b2-84036908df7c","Type":"ContainerStarted","Data":"b1f68ac0f15ff0208038638b270c7c7bcd4e628e99c4c1efc1067842ed7c5437"} Oct 30 05:46:17.746583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.746555 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" event={"ID":"27f64629-383b-4efd-95b2-84036908df7c","Type":"ContainerStarted","Data":"8cf5920ad3b3a4e44789fb6972fc18d49fb73d6d5307614e365d5926e8de47db"} Oct 30 05:46:17.971555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:17.969689 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:18.974550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:18.971639 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:46:18.974550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:18.971716 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:46:19.975559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:19.973936 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:46:19.975559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:19.974042 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:46:20.973553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:20.972926 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:46:20.973553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:20.973028 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:46:21.967553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:21.967023 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:46:21.967553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:21.967088 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:46:22.579575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.579342 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podStartSLOduration=6.579324672 podStartE2EDuration="6.579324672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:17.808610334 +0000 UTC m=+884.384297388" watchObservedRunningTime="2026-10-30 05:46:22.579324672 +0000 UTC m=+889.155011727" Oct 30 05:46:22.581493 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.581472 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh"] Oct 30 05:46:22.585496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.585478 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.614434 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.614399 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:22.614661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.614456 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:22.614661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.614478 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:22.614661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.614514 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:46:22.617823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.617795 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh"] Oct 30 05:46:22.713039 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.713000 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.713214 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.713126 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.713313 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.713297 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sqhx\" (UniqueName: \"kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.780984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.780931 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k"] Oct 30 05:46:22.791410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.791382 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k"] Oct 30 05:46:22.791588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.791482 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:22.814701 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.814671 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.814823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.814708 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.814823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.814791 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sqhx\" (UniqueName: \"kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.815869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.815847 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.817328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.817305 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.840622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.840566 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sqhx\" (UniqueName: \"kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx\") pod \"maas-api-revoke-sub-3552c301-9t4wh\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.900461 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.900423 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:22.916215 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.916186 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:22.916410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.916387 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:22.916512 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.916494 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ps4cr\" (UniqueName: \"kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:22.982154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.981651 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 05:46:22.982154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:22.981715 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 05:46:23.018004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.017854 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.018004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.017996 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.018256 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.018057 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ps4cr\" (UniqueName: \"kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.018887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.018712 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.021228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.021203 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.042381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.042238 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ps4cr\" (UniqueName: \"kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr\") pod \"maas-api-revoke-sub-cbf9c38e-sbv5k\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.106666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.106582 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:23.237256 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.236971 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh"] Oct 30 05:46:23.239584 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:23.239540 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb24a52da_4a84_45aa_bd96_e3c6f53758d2.slice/crio-deb773e1c2674b3154544a5b23c326d6db55df87ab581ce5f63d01dcef2a0fe1 WatchSource:0}: Error finding container deb773e1c2674b3154544a5b23c326d6db55df87ab581ce5f63d01dcef2a0fe1: Status 404 returned error can't find the container with id deb773e1c2674b3154544a5b23c326d6db55df87ab581ce5f63d01dcef2a0fe1 Oct 30 05:46:23.440723 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.440697 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k"] Oct 30 05:46:23.443066 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:23.443041 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4995005_531f_4f80_91f5_e1d240dc4404.slice/crio-de1cf39e48c82589570db8f5bf363c87e5e3566d7936e2a62c7054d61862f4a0 WatchSource:0}: Error finding container de1cf39e48c82589570db8f5bf363c87e5e3566d7936e2a62c7054d61862f4a0: Status 404 returned error can't find the container with id de1cf39e48c82589570db8f5bf363c87e5e3566d7936e2a62c7054d61862f4a0 Oct 30 05:46:23.814739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.814696 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" event={"ID":"b24a52da-4a84-45aa-bd96-e3c6f53758d2","Type":"ContainerStarted","Data":"3e9add7862d969138dbc42f0b37677e37d67f0f59a1baa7dacbbcc0d6111fa6b"} Oct 30 05:46:23.814924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.814758 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" event={"ID":"b24a52da-4a84-45aa-bd96-e3c6f53758d2","Type":"ContainerStarted","Data":"deb773e1c2674b3154544a5b23c326d6db55df87ab581ce5f63d01dcef2a0fe1"} Oct 30 05:46:23.816586 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.816558 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" event={"ID":"d4995005-531f-4f80-91f5-e1d240dc4404","Type":"ContainerStarted","Data":"ea02ab23d50b5e175f9ffd2eba9e6e1f6b70d667ff74a1580457bb108ebd31ba"} Oct 30 05:46:23.816743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.816605 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" event={"ID":"d4995005-531f-4f80-91f5-e1d240dc4404","Type":"ContainerStarted","Data":"de1cf39e48c82589570db8f5bf363c87e5e3566d7936e2a62c7054d61862f4a0"} Oct 30 05:46:23.858082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.858031 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" podStartSLOduration=1.858006732 podStartE2EDuration="1.858006732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:23.85595149 +0000 UTC m=+890.431638546" watchObservedRunningTime="2026-10-30 05:46:23.858006732 +0000 UTC m=+890.433693789" Oct 30 05:46:23.900895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:23.900830 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" podStartSLOduration=1.900810855 podStartE2EDuration="1.900810855s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:23.8974247 +0000 UTC m=+890.473111755" watchObservedRunningTime="2026-10-30 05:46:23.900810855 +0000 UTC m=+890.476497912" Oct 30 05:46:24.093133 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.093049 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:24.093587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.093474 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:24.095237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.095218 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 05:46:24.823470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.823439 5565 generic.go:356] "Generic (PLEG): container finished" podID="b24a52da-4a84-45aa-bd96-e3c6f53758d2" containerID="3e9add7862d969138dbc42f0b37677e37d67f0f59a1baa7dacbbcc0d6111fa6b" exitCode=0 Oct 30 05:46:24.823667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.823549 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" event={"ID":"b24a52da-4a84-45aa-bd96-e3c6f53758d2","Type":"ContainerDied","Data":"3e9add7862d969138dbc42f0b37677e37d67f0f59a1baa7dacbbcc0d6111fa6b"} Oct 30 05:46:24.825092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.825066 5565 generic.go:356] "Generic (PLEG): container finished" podID="d4995005-531f-4f80-91f5-e1d240dc4404" containerID="ea02ab23d50b5e175f9ffd2eba9e6e1f6b70d667ff74a1580457bb108ebd31ba" exitCode=0 Oct 30 05:46:24.825191 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:24.825123 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" event={"ID":"d4995005-531f-4f80-91f5-e1d240dc4404","Type":"ContainerDied","Data":"ea02ab23d50b5e175f9ffd2eba9e6e1f6b70d667ff74a1580457bb108ebd31ba"} Oct 30 05:46:26.001268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.000698 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:26.034398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.034372 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:26.155477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155392 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca\") pod \"d4995005-531f-4f80-91f5-e1d240dc4404\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " Oct 30 05:46:26.155477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155435 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca\") pod \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " Oct 30 05:46:26.155726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155548 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr\" (UniqueName: \"kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr\") pod \"d4995005-531f-4f80-91f5-e1d240dc4404\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " Oct 30 05:46:26.155726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155590 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca\") pod \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " Oct 30 05:46:26.155726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155654 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca\") pod \"d4995005-531f-4f80-91f5-e1d240dc4404\" (UID: \"d4995005-531f-4f80-91f5-e1d240dc4404\") " Oct 30 05:46:26.155892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155729 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx\" (UniqueName: \"kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx\") pod \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\" (UID: \"b24a52da-4a84-45aa-bd96-e3c6f53758d2\") " Oct 30 05:46:26.155892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.155870 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca" pod "b24a52da-4a84-45aa-bd96-e3c6f53758d2" (UID: "b24a52da-4a84-45aa-bd96-e3c6f53758d2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:26.156087 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.156061 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca" pod "d4995005-531f-4f80-91f5-e1d240dc4404" (UID: "d4995005-531f-4f80-91f5-e1d240dc4404"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:26.156731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.156479 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:26.156731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.156508 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:26.157843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.157821 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr" pod "d4995005-531f-4f80-91f5-e1d240dc4404" (UID: "d4995005-531f-4f80-91f5-e1d240dc4404"). InnerVolumeSpecName "kube-api-access-ps4cr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:26.158086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.158063 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca" pod "b24a52da-4a84-45aa-bd96-e3c6f53758d2" (UID: "b24a52da-4a84-45aa-bd96-e3c6f53758d2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:26.158206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.158093 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca" pod "d4995005-531f-4f80-91f5-e1d240dc4404" (UID: "d4995005-531f-4f80-91f5-e1d240dc4404"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:26.158206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.158157 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx" pod "b24a52da-4a84-45aa-bd96-e3c6f53758d2" (UID: "b24a52da-4a84-45aa-bd96-e3c6f53758d2"). InnerVolumeSpecName "kube-api-access-8sqhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:26.162198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162171 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq"] Oct 30 05:46:26.162724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162709 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d4995005-531f-4f80-91f5-e1d240dc4404" containerName="revoke-keys" Oct 30 05:46:26.162821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162725 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4995005-531f-4f80-91f5-e1d240dc4404" containerName="revoke-keys" Oct 30 05:46:26.162821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162758 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b24a52da-4a84-45aa-bd96-e3c6f53758d2" containerName="revoke-keys" Oct 30 05:46:26.162821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162763 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b24a52da-4a84-45aa-bd96-e3c6f53758d2" containerName="revoke-keys" Oct 30 05:46:26.162979 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162881 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b24a52da-4a84-45aa-bd96-e3c6f53758d2" containerName="revoke-keys" Oct 30 05:46:26.162979 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.162890 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4995005-531f-4f80-91f5-e1d240dc4404" containerName="revoke-keys" Oct 30 05:46:26.166741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.166722 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.188175 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.188153 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq"] Oct 30 05:46:26.258174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.258063 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ps4cr\" (UniqueName: \"kubernetes.io/projected/d4995005-531f-4f80-91f5-e1d240dc4404-kube-api-access-ps4cr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:26.258174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.258097 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b24a52da-4a84-45aa-bd96-e3c6f53758d2-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:26.258174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.258112 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sqhx\" (UniqueName: \"kubernetes.io/projected/b24a52da-4a84-45aa-bd96-e3c6f53758d2-kube-api-access-8sqhx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:26.258174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.258129 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4995005-531f-4f80-91f5-e1d240dc4404-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:26.358792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.358760 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.358943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.358927 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kgzw\" (UniqueName: \"kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.358983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.358953 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.460010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.459930 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kgzw\" (UniqueName: \"kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.460010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.459964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.460228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.460026 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.460661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.460634 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.462294 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.462270 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.488616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.488595 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kgzw\" (UniqueName: \"kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw\") pod \"maas-api-revoke-sub-efd5a661-flzcq\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.783145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.783116 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:26.839965 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.839931 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" event={"ID":"b24a52da-4a84-45aa-bd96-e3c6f53758d2","Type":"ContainerDied","Data":"deb773e1c2674b3154544a5b23c326d6db55df87ab581ce5f63d01dcef2a0fe1"} Oct 30 05:46:26.840069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.839973 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh" Oct 30 05:46:26.840069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.839996 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb773e1c2674b3154544a5b23c326d6db55df87ab581ce5f63d01dcef2a0fe1" Oct 30 05:46:26.846628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.846599 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" event={"ID":"d4995005-531f-4f80-91f5-e1d240dc4404","Type":"ContainerDied","Data":"de1cf39e48c82589570db8f5bf363c87e5e3566d7936e2a62c7054d61862f4a0"} Oct 30 05:46:26.846758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.846651 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="de1cf39e48c82589570db8f5bf363c87e5e3566d7936e2a62c7054d61862f4a0" Oct 30 05:46:26.846758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:26.846714 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k" Oct 30 05:46:27.086424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:27.086397 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq"] Oct 30 05:46:27.088397 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:27.088372 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81bc1317_6af2_4689_99bb_4d696b00d08c.slice/crio-27246aeb120f4649ffbe7d0fcf72dc6e2e1234cabf170bda4ce9204e25e906c1 WatchSource:0}: Error finding container 27246aeb120f4649ffbe7d0fcf72dc6e2e1234cabf170bda4ce9204e25e906c1: Status 404 returned error can't find the container with id 27246aeb120f4649ffbe7d0fcf72dc6e2e1234cabf170bda4ce9204e25e906c1 Oct 30 05:46:27.853645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:27.853613 5565 generic.go:356] "Generic (PLEG): container finished" podID="81bc1317-6af2-4689-99bb-4d696b00d08c" containerID="3e43a916784ee1f27301d640534e15ce1303db6a32e4af6fac62edbddf93b6c0" exitCode=0 Oct 30 05:46:27.853781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:27.853697 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" event={"ID":"81bc1317-6af2-4689-99bb-4d696b00d08c","Type":"ContainerDied","Data":"3e43a916784ee1f27301d640534e15ce1303db6a32e4af6fac62edbddf93b6c0"} Oct 30 05:46:27.853781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:27.853761 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" event={"ID":"81bc1317-6af2-4689-99bb-4d696b00d08c","Type":"ContainerStarted","Data":"27246aeb120f4649ffbe7d0fcf72dc6e2e1234cabf170bda4ce9204e25e906c1"} Oct 30 05:46:28.368972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.368938 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:46:28.582053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.582027 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv"] Oct 30 05:46:28.585950 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.585933 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.607338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.607312 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv"] Oct 30 05:46:28.678971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.678901 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.679082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.679004 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmw92\" (UniqueName: \"kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.679082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.679055 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.780132 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.780106 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmw92\" (UniqueName: \"kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.780260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.780150 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.780260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.780218 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.780838 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.780820 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.782478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.782458 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.804180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.804162 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmw92\" (UniqueName: \"kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92\") pod \"maas-api-revoke-sub-fe82a2be-9v9rv\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:28.898695 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:28.898658 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:29.004016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.003994 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:29.083804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.083776 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca\") pod \"81bc1317-6af2-4689-99bb-4d696b00d08c\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " Oct 30 05:46:29.083962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.083901 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca\") pod \"81bc1317-6af2-4689-99bb-4d696b00d08c\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " Oct 30 05:46:29.084014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.083962 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw\" (UniqueName: \"kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw\") pod \"81bc1317-6af2-4689-99bb-4d696b00d08c\" (UID: \"81bc1317-6af2-4689-99bb-4d696b00d08c\") " Oct 30 05:46:29.084258 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.084213 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca" pod "81bc1317-6af2-4689-99bb-4d696b00d08c" (UID: "81bc1317-6af2-4689-99bb-4d696b00d08c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:29.084359 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.084304 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:29.085894 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.085869 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca" pod "81bc1317-6af2-4689-99bb-4d696b00d08c" (UID: "81bc1317-6af2-4689-99bb-4d696b00d08c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:29.085971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.085925 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw" pod "81bc1317-6af2-4689-99bb-4d696b00d08c" (UID: "81bc1317-6af2-4689-99bb-4d696b00d08c"). InnerVolumeSpecName "kube-api-access-7kgzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:29.185835 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.185809 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kgzw\" (UniqueName: \"kubernetes.io/projected/81bc1317-6af2-4689-99bb-4d696b00d08c-kube-api-access-7kgzw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:29.185962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.185843 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81bc1317-6af2-4689-99bb-4d696b00d08c-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:29.214782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.214764 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv"] Oct 30 05:46:29.215688 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:29.215625 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13a086c_9037_492c_8f42_a613c09dfbec.slice/crio-9d266676bd692adc81ac6fd222afc4765c03a5a06bfc8b056c1905e0ded0973b WatchSource:0}: Error finding container 9d266676bd692adc81ac6fd222afc4765c03a5a06bfc8b056c1905e0ded0973b: Status 404 returned error can't find the container with id 9d266676bd692adc81ac6fd222afc4765c03a5a06bfc8b056c1905e0ded0973b Oct 30 05:46:29.886088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.885992 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" event={"ID":"c13a086c-9037-492c-8f42-a613c09dfbec","Type":"ContainerStarted","Data":"092973f40dbb89d99b035cdbd1f0fb9e99c912aa0c1e9c607acba796364315e4"} Oct 30 05:46:29.886687 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.886647 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" event={"ID":"c13a086c-9037-492c-8f42-a613c09dfbec","Type":"ContainerStarted","Data":"9d266676bd692adc81ac6fd222afc4765c03a5a06bfc8b056c1905e0ded0973b"} Oct 30 05:46:29.889486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.889461 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" event={"ID":"81bc1317-6af2-4689-99bb-4d696b00d08c","Type":"ContainerDied","Data":"27246aeb120f4649ffbe7d0fcf72dc6e2e1234cabf170bda4ce9204e25e906c1"} Oct 30 05:46:29.889689 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.889671 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27246aeb120f4649ffbe7d0fcf72dc6e2e1234cabf170bda4ce9204e25e906c1" Oct 30 05:46:29.889830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.889817 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq" Oct 30 05:46:29.933553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:29.933151 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" podStartSLOduration=1.933131672 podStartE2EDuration="1.933131672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:29.929143531 +0000 UTC m=+896.504830589" watchObservedRunningTime="2026-10-30 05:46:29.933131672 +0000 UTC m=+896.508818727" Oct 30 05:46:30.896827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:30.896791 5565 generic.go:356] "Generic (PLEG): container finished" podID="c13a086c-9037-492c-8f42-a613c09dfbec" containerID="092973f40dbb89d99b035cdbd1f0fb9e99c912aa0c1e9c607acba796364315e4" exitCode=0 Oct 30 05:46:30.897194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:30.896868 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" event={"ID":"c13a086c-9037-492c-8f42-a613c09dfbec","Type":"ContainerDied","Data":"092973f40dbb89d99b035cdbd1f0fb9e99c912aa0c1e9c607acba796364315e4"} Oct 30 05:46:32.050702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.050678 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:32.143589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.143562 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca\") pod \"c13a086c-9037-492c-8f42-a613c09dfbec\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " Oct 30 05:46:32.143724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.143604 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92\" (UniqueName: \"kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92\") pod \"c13a086c-9037-492c-8f42-a613c09dfbec\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " Oct 30 05:46:32.143814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.143796 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca\") pod \"c13a086c-9037-492c-8f42-a613c09dfbec\" (UID: \"c13a086c-9037-492c-8f42-a613c09dfbec\") " Oct 30 05:46:32.144138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.144106 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca" pod "c13a086c-9037-492c-8f42-a613c09dfbec" (UID: "c13a086c-9037-492c-8f42-a613c09dfbec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:32.144265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.144232 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:32.146101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.146080 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92" pod "c13a086c-9037-492c-8f42-a613c09dfbec" (UID: "c13a086c-9037-492c-8f42-a613c09dfbec"). InnerVolumeSpecName "kube-api-access-bmw92". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:32.146101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.146084 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca" pod "c13a086c-9037-492c-8f42-a613c09dfbec" (UID: "c13a086c-9037-492c-8f42-a613c09dfbec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:32.245106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.245069 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c13a086c-9037-492c-8f42-a613c09dfbec-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:32.245106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.245098 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmw92\" (UniqueName: \"kubernetes.io/projected/c13a086c-9037-492c-8f42-a613c09dfbec-kube-api-access-bmw92\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:32.909107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.909077 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx"] Oct 30 05:46:32.909607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.909588 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" Oct 30 05:46:32.909882 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.909865 5565 cpu_manager.go:414] "containerMap: removing container" podUID="81bc1317-6af2-4689-99bb-4d696b00d08c" containerName="revoke-keys" Oct 30 05:46:32.909949 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.909885 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81bc1317-6af2-4689-99bb-4d696b00d08c" containerName="revoke-keys" Oct 30 05:46:32.909949 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.909901 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c13a086c-9037-492c-8f42-a613c09dfbec" containerName="revoke-keys" Oct 30 05:46:32.909949 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.909908 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c13a086c-9037-492c-8f42-a613c09dfbec" containerName="revoke-keys" Oct 30 05:46:32.910123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.910108 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="81bc1317-6af2-4689-99bb-4d696b00d08c" containerName="revoke-keys" Oct 30 05:46:32.910178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.910126 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c13a086c-9037-492c-8f42-a613c09dfbec" containerName="revoke-keys" Oct 30 05:46:32.913997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.913974 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv" event={"ID":"c13a086c-9037-492c-8f42-a613c09dfbec","Type":"ContainerDied","Data":"9d266676bd692adc81ac6fd222afc4765c03a5a06bfc8b056c1905e0ded0973b"} Oct 30 05:46:32.914095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.914011 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d266676bd692adc81ac6fd222afc4765c03a5a06bfc8b056c1905e0ded0973b" Oct 30 05:46:32.914130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.914097 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:32.935151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.935118 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx"] Oct 30 05:46:32.944829 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.944797 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:32.944829 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.944795 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:32.944829 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.944796 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:32.945017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.944864 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:46:32.956795 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.956769 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnx7f\" (UniqueName: \"kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:32.956936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.956814 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:32.956996 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:32.956930 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.057727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.057694 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnx7f\" (UniqueName: \"kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.057727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.057735 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.058276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.057772 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.058409 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.058387 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.060097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.060072 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.082632 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.082602 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnx7f\" (UniqueName: \"kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f\") pod \"maas-api-revoke-sub-9d7a17fb-z4mkx\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.229069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.228974 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:33.537735 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.537707 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx"] Oct 30 05:46:33.538331 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:33.538305 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73e71e68_278c_4423_b8e8_a8b7db8d4553.slice/crio-21b87e13d8fe09ac2471439920d3bef3ec42b44b6ed55e35fe9a80a0d9f9b7be WatchSource:0}: Error finding container 21b87e13d8fe09ac2471439920d3bef3ec42b44b6ed55e35fe9a80a0d9f9b7be: Status 404 returned error can't find the container with id 21b87e13d8fe09ac2471439920d3bef3ec42b44b6ed55e35fe9a80a0d9f9b7be Oct 30 05:46:33.917264 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:33.917229 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" event={"ID":"73e71e68-278c-4423-b8e8-a8b7db8d4553","Type":"ContainerStarted","Data":"21b87e13d8fe09ac2471439920d3bef3ec42b44b6ed55e35fe9a80a0d9f9b7be"} Oct 30 05:46:34.924424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:34.924387 5565 generic.go:356] "Generic (PLEG): container finished" podID="73e71e68-278c-4423-b8e8-a8b7db8d4553" containerID="252ecb187a2a849e72cb5b28253d0817ae70ec96a0b41b2d67bc7011a6daf2d1" exitCode=0 Oct 30 05:46:34.924968 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:34.924471 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" event={"ID":"73e71e68-278c-4423-b8e8-a8b7db8d4553","Type":"ContainerDied","Data":"252ecb187a2a849e72cb5b28253d0817ae70ec96a0b41b2d67bc7011a6daf2d1"} Oct 30 05:46:36.075275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.075249 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:36.190057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.189968 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca\") pod \"73e71e68-278c-4423-b8e8-a8b7db8d4553\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " Oct 30 05:46:36.190057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.190025 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca\") pod \"73e71e68-278c-4423-b8e8-a8b7db8d4553\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " Oct 30 05:46:36.190057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.190055 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f\" (UniqueName: \"kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f\") pod \"73e71e68-278c-4423-b8e8-a8b7db8d4553\" (UID: \"73e71e68-278c-4423-b8e8-a8b7db8d4553\") " Oct 30 05:46:36.190405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.190375 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca" pod "73e71e68-278c-4423-b8e8-a8b7db8d4553" (UID: "73e71e68-278c-4423-b8e8-a8b7db8d4553"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:36.192211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.192174 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f" pod "73e71e68-278c-4423-b8e8-a8b7db8d4553" (UID: "73e71e68-278c-4423-b8e8-a8b7db8d4553"). InnerVolumeSpecName "kube-api-access-dnx7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:36.192360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.192336 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca" pod "73e71e68-278c-4423-b8e8-a8b7db8d4553" (UID: "73e71e68-278c-4423-b8e8-a8b7db8d4553"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:36.290951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.290918 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:36.290951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.290949 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73e71e68-278c-4423-b8e8-a8b7db8d4553-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:36.291156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.290962 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnx7f\" (UniqueName: \"kubernetes.io/projected/73e71e68-278c-4423-b8e8-a8b7db8d4553-kube-api-access-dnx7f\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:36.557129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.557097 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s"] Oct 30 05:46:36.557684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.557668 5565 cpu_manager.go:414] "containerMap: removing container" podUID="73e71e68-278c-4423-b8e8-a8b7db8d4553" containerName="revoke-keys" Oct 30 05:46:36.557736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.557688 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73e71e68-278c-4423-b8e8-a8b7db8d4553" containerName="revoke-keys" Oct 30 05:46:36.557805 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.557794 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="73e71e68-278c-4423-b8e8-a8b7db8d4553" containerName="revoke-keys" Oct 30 05:46:36.561391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.561366 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.582771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.582731 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s"] Oct 30 05:46:36.701343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.701303 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.701516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.701446 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.701516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.701481 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7znfs\" (UniqueName: \"kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.802290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.802256 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.802461 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.802378 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.802461 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.802405 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7znfs\" (UniqueName: \"kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.802968 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.802944 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.804713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.804693 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.826320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.826261 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7znfs\" (UniqueName: \"kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs\") pod \"maas-api-revoke-sub-0bee557b-ddm7s\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.875590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.875556 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:36.945185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.945160 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" Oct 30 05:46:36.945333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.945213 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx" event={"ID":"73e71e68-278c-4423-b8e8-a8b7db8d4553","Type":"ContainerDied","Data":"21b87e13d8fe09ac2471439920d3bef3ec42b44b6ed55e35fe9a80a0d9f9b7be"} Oct 30 05:46:36.945333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:36.945275 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21b87e13d8fe09ac2471439920d3bef3ec42b44b6ed55e35fe9a80a0d9f9b7be" Oct 30 05:46:37.183223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:37.183199 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s"] Oct 30 05:46:37.184573 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:37.184544 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09979dbb_f136_4d63_a622_8ddbf72b0d1f.slice/crio-23a5dea5859a8b790707714a97f0fd17f906826dbac1f35f7a6bdeb9cebfdf68 WatchSource:0}: Error finding container 23a5dea5859a8b790707714a97f0fd17f906826dbac1f35f7a6bdeb9cebfdf68: Status 404 returned error can't find the container with id 23a5dea5859a8b790707714a97f0fd17f906826dbac1f35f7a6bdeb9cebfdf68 Oct 30 05:46:37.952640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:37.952606 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" event={"ID":"09979dbb-f136-4d63-a622-8ddbf72b0d1f","Type":"ContainerStarted","Data":"cd2e6b28a81a4d9b6f05cc8a22d4a17eff485f87c5682ee3364f51931b7c33ab"} Oct 30 05:46:37.952819 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:37.952662 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" event={"ID":"09979dbb-f136-4d63-a622-8ddbf72b0d1f","Type":"ContainerStarted","Data":"23a5dea5859a8b790707714a97f0fd17f906826dbac1f35f7a6bdeb9cebfdf68"} Oct 30 05:46:37.994985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:37.994931 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" podStartSLOduration=1.9949168 podStartE2EDuration="1.9949168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:37.993493473 +0000 UTC m=+904.569180528" watchObservedRunningTime="2026-10-30 05:46:37.9949168 +0000 UTC m=+904.570603885" Oct 30 05:46:38.962228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:38.962193 5565 generic.go:356] "Generic (PLEG): container finished" podID="09979dbb-f136-4d63-a622-8ddbf72b0d1f" containerID="cd2e6b28a81a4d9b6f05cc8a22d4a17eff485f87c5682ee3364f51931b7c33ab" exitCode=0 Oct 30 05:46:38.962676 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:38.962274 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" event={"ID":"09979dbb-f136-4d63-a622-8ddbf72b0d1f","Type":"ContainerDied","Data":"cd2e6b28a81a4d9b6f05cc8a22d4a17eff485f87c5682ee3364f51931b7c33ab"} Oct 30 05:46:39.231004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.230916 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg"] Oct 30 05:46:39.234975 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.234950 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.257157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.257128 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg"] Oct 30 05:46:39.330114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.330082 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.330288 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.330254 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcb8k\" (UniqueName: \"kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.330288 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.330278 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.431603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.431572 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcb8k\" (UniqueName: \"kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.431603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.431607 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.431834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.431648 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.432232 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.432200 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.433981 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.433955 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.455942 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.455918 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcb8k\" (UniqueName: \"kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k\") pod \"maas-api-revoke-sub-8851028d-vdzmg\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.547814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.547783 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:39.856658 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.856629 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg"] Oct 30 05:46:39.857827 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:39.857791 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1300fbbc_7ab5_47fb_825a_3384ff24a461.slice/crio-430baa42d54eeef62ca18618a4903f8f5629b17cde8a0a6fd95cb529251b6108 WatchSource:0}: Error finding container 430baa42d54eeef62ca18618a4903f8f5629b17cde8a0a6fd95cb529251b6108: Status 404 returned error can't find the container with id 430baa42d54eeef62ca18618a4903f8f5629b17cde8a0a6fd95cb529251b6108 Oct 30 05:46:39.969734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:39.969698 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" event={"ID":"1300fbbc-7ab5-47fb-825a-3384ff24a461","Type":"ContainerStarted","Data":"430baa42d54eeef62ca18618a4903f8f5629b17cde8a0a6fd95cb529251b6108"} Oct 30 05:46:40.088977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.088944 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:40.240360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.240333 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca\") pod \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " Oct 30 05:46:40.240598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.240385 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs\" (UniqueName: \"kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs\") pod \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " Oct 30 05:46:40.240598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.240492 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca\") pod \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\" (UID: \"09979dbb-f136-4d63-a622-8ddbf72b0d1f\") " Oct 30 05:46:40.240779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.240755 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca" pod "09979dbb-f136-4d63-a622-8ddbf72b0d1f" (UID: "09979dbb-f136-4d63-a622-8ddbf72b0d1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:40.242666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.242645 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca" pod "09979dbb-f136-4d63-a622-8ddbf72b0d1f" (UID: "09979dbb-f136-4d63-a622-8ddbf72b0d1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:40.242740 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.242689 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs" pod "09979dbb-f136-4d63-a622-8ddbf72b0d1f" (UID: "09979dbb-f136-4d63-a622-8ddbf72b0d1f"). InnerVolumeSpecName "kube-api-access-7znfs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:40.287789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.287754 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg"] Oct 30 05:46:40.288607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.288584 5565 cpu_manager.go:414] "containerMap: removing container" podUID="09979dbb-f136-4d63-a622-8ddbf72b0d1f" containerName="revoke-keys" Oct 30 05:46:40.288693 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.288608 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09979dbb-f136-4d63-a622-8ddbf72b0d1f" containerName="revoke-keys" Oct 30 05:46:40.288792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.288780 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="09979dbb-f136-4d63-a622-8ddbf72b0d1f" containerName="revoke-keys" Oct 30 05:46:40.292737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.292713 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.321559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.321379 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-698ab2\"/\"distinct-w4-698ab2-kserve-self-signed-certs\"" Oct 30 05:46:40.324067 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.324040 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg"] Oct 30 05:46:40.341498 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.341467 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:40.341498 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.341493 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/09979dbb-f136-4d63-a622-8ddbf72b0d1f-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:40.341699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.341505 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7znfs\" (UniqueName: \"kubernetes.io/projected/09979dbb-f136-4d63-a622-8ddbf72b0d1f-kube-api-access-7znfs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:40.442612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.442481 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.442612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.442556 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.442750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.442633 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.442750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.442707 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.442922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.442893 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlzk5\" (UniqueName: \"kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.443016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.443005 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.544620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.544584 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.544791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.544684 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlzk5\" (UniqueName: \"kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.544791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.544748 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.544901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.544798 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.544901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.544864 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.544901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.544892 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.545094 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.545025 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.545150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.545109 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.545222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.545207 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.546979 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.546949 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.547236 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.547220 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.571288 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.571224 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlzk5\" (UniqueName: \"kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5\") pod \"distinct-w4-698ab2-kserve-57d58c647f-mxbzg\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.606851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.606812 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:40.920297 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.920271 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg"] Oct 30 05:46:40.921486 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:40.921457 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19085093_07d1_4cf4_813f_04f6e159b7dc.slice/crio-9b34af30277e2c9df5eb0753858a0ba3e30f41631677fcede313f486445cb0df WatchSource:0}: Error finding container 9b34af30277e2c9df5eb0753858a0ba3e30f41631677fcede313f486445cb0df: Status 404 returned error can't find the container with id 9b34af30277e2c9df5eb0753858a0ba3e30f41631677fcede313f486445cb0df Oct 30 05:46:40.977253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.977233 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" Oct 30 05:46:40.977745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.977249 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s" event={"ID":"09979dbb-f136-4d63-a622-8ddbf72b0d1f","Type":"ContainerDied","Data":"23a5dea5859a8b790707714a97f0fd17f906826dbac1f35f7a6bdeb9cebfdf68"} Oct 30 05:46:40.977745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.977310 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23a5dea5859a8b790707714a97f0fd17f906826dbac1f35f7a6bdeb9cebfdf68" Oct 30 05:46:40.979241 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.979220 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" event={"ID":"19085093-07d1-4cf4-813f-04f6e159b7dc","Type":"ContainerStarted","Data":"9b34af30277e2c9df5eb0753858a0ba3e30f41631677fcede313f486445cb0df"} Oct 30 05:46:40.980977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.980958 5565 generic.go:356] "Generic (PLEG): container finished" podID="1300fbbc-7ab5-47fb-825a-3384ff24a461" containerID="fca099f7856d11d6d46bc8caea0fa111d38700b82400ba17a7b13ee0b5849b05" exitCode=0 Oct 30 05:46:40.981082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:40.981037 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" event={"ID":"1300fbbc-7ab5-47fb-825a-3384ff24a461","Type":"ContainerDied","Data":"fca099f7856d11d6d46bc8caea0fa111d38700b82400ba17a7b13ee0b5849b05"} Oct 30 05:46:41.988352 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:41.988315 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" event={"ID":"19085093-07d1-4cf4-813f-04f6e159b7dc","Type":"ContainerStarted","Data":"2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243"} Oct 30 05:46:42.053582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.043892 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podStartSLOduration=2.043870591 podStartE2EDuration="2.043870591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:42.040914945 +0000 UTC m=+908.616601999" watchObservedRunningTime="2026-10-30 05:46:42.043870591 +0000 UTC m=+908.619557651" Oct 30 05:46:42.153500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.153463 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:42.266609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.266573 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca\") pod \"1300fbbc-7ab5-47fb-825a-3384ff24a461\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " Oct 30 05:46:42.266779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.266761 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca\") pod \"1300fbbc-7ab5-47fb-825a-3384ff24a461\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " Oct 30 05:46:42.266821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.266801 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k\" (UniqueName: \"kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k\") pod \"1300fbbc-7ab5-47fb-825a-3384ff24a461\" (UID: \"1300fbbc-7ab5-47fb-825a-3384ff24a461\") " Oct 30 05:46:42.267120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.267096 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca" pod "1300fbbc-7ab5-47fb-825a-3384ff24a461" (UID: "1300fbbc-7ab5-47fb-825a-3384ff24a461"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:42.267306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.267289 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:42.269181 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.269148 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k" pod "1300fbbc-7ab5-47fb-825a-3384ff24a461" (UID: "1300fbbc-7ab5-47fb-825a-3384ff24a461"). InnerVolumeSpecName "kube-api-access-zcb8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:42.269275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.269157 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca" pod "1300fbbc-7ab5-47fb-825a-3384ff24a461" (UID: "1300fbbc-7ab5-47fb-825a-3384ff24a461"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:42.368850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.368826 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1300fbbc-7ab5-47fb-825a-3384ff24a461-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:42.368850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.368849 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcb8k\" (UniqueName: \"kubernetes.io/projected/1300fbbc-7ab5-47fb-825a-3384ff24a461-kube-api-access-zcb8k\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:42.607998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:42.607903 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:43.023552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:43.012571 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" Oct 30 05:46:43.030944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:43.030692 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg" event={"ID":"1300fbbc-7ab5-47fb-825a-3384ff24a461","Type":"ContainerDied","Data":"430baa42d54eeef62ca18618a4903f8f5629b17cde8a0a6fd95cb529251b6108"} Oct 30 05:46:43.030944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:43.030756 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="430baa42d54eeef62ca18618a4903f8f5629b17cde8a0a6fd95cb529251b6108" Oct 30 05:46:47.331654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.331562 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7"] Oct 30 05:46:47.332116 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.332102 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1300fbbc-7ab5-47fb-825a-3384ff24a461" containerName="revoke-keys" Oct 30 05:46:47.332187 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.332118 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1300fbbc-7ab5-47fb-825a-3384ff24a461" containerName="revoke-keys" Oct 30 05:46:47.332231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.332220 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1300fbbc-7ab5-47fb-825a-3384ff24a461" containerName="revoke-keys" Oct 30 05:46:47.335923 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.335899 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.359586 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359554 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6mvw\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.359736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359600 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.359821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359727 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.359821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359789 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.359918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359833 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.359918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359891 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.360001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359918 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.360001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.359940 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.360071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.360052 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.360104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.360086 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-e7a506-openshift-default-dockercfg-552kl\"" Oct 30 05:46:47.374712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.374687 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7"] Oct 30 05:46:47.461823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.461789 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6mvw\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.461993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.461856 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.461993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.461895 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.461993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.461939 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.461993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.461981 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.462216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462011 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.462216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462043 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.462216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462069 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.462216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462127 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.462830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462791 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.463061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462815 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.463177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.462832 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.463177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.463031 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.463177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.463118 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.464592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.464547 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.464811 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.464788 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.485252 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.485226 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.485632 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.485613 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6mvw\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.531674 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:46:47.531643 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:46:47.649790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.649704 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:47.976204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.976168 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7"] Oct 30 05:46:47.976822 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:47.976796 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588bcdf5_b140_4542_9d75_30ca04152ab8.slice/crio-0bb5db0d9298e80e9e8a163235092cf370d4d9f951c2ecd779580f53f088b039 WatchSource:0}: Error finding container 0bb5db0d9298e80e9e8a163235092cf370d4d9f951c2ecd779580f53f088b039: Status 404 returned error can't find the container with id 0bb5db0d9298e80e9e8a163235092cf370d4d9f951c2ecd779580f53f088b039 Oct 30 05:46:47.979090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.979057 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:47.979174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.979120 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:47.979174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:47.979148 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:46:48.051462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:48.051434 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" event={"ID":"588bcdf5-b140-4542-9d75-30ca04152ab8","Type":"ContainerStarted","Data":"0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86"} Oct 30 05:46:48.051573 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:48.051493 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" event={"ID":"588bcdf5-b140-4542-9d75-30ca04152ab8","Type":"ContainerStarted","Data":"0bb5db0d9298e80e9e8a163235092cf370d4d9f951c2ecd779580f53f088b039"} Oct 30 05:46:48.101914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:48.101862 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podStartSLOduration=1.10184795 podStartE2EDuration="1.10184795s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:48.099280488 +0000 UTC m=+914.674967543" watchObservedRunningTime="2026-10-30 05:46:48.10184795 +0000 UTC m=+914.677535003" Oct 30 05:46:48.649880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:48.649832 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:49.625743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.625699 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd"] Oct 30 05:46:49.629920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.629896 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.646790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.646754 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:46:49.647050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.647011 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:46:49.651972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.651939 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:46:49.652360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.652001 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:46:49.682552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.673918 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd"] Oct 30 05:46:49.683738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.683702 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:46:49.683738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.683702 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:46:49.690477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.690430 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.690747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.690717 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.690834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.690810 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp2sh\" (UniqueName: \"kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.791592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.791508 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.791799 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.791605 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp2sh\" (UniqueName: \"kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.791799 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.791678 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.792357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.792333 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.794096 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.794069 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.816586 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.816545 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp2sh\" (UniqueName: \"kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh\") pod \"maas-api-revoke-sub-274cd53a-f8chd\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:49.945582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:49.945480 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:50.181114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.178206 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9"] Oct 30 05:46:50.183263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.183222 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9"] Oct 30 05:46:50.183421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.183363 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.196302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.196200 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr2kq\" (UniqueName: \"kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.196477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.196340 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.196477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.196438 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.281453 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.281415 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd"] Oct 30 05:46:50.282815 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:50.282782 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe5f6735_b306_4bc1_af35_741411c0b65e.slice/crio-fb4bb8910ebbf817611efa4680e9ae6e289bf4d41c263c5d4a0aa4afec48cf51 WatchSource:0}: Error finding container fb4bb8910ebbf817611efa4680e9ae6e289bf4d41c263c5d4a0aa4afec48cf51: Status 404 returned error can't find the container with id fb4bb8910ebbf817611efa4680e9ae6e289bf4d41c263c5d4a0aa4afec48cf51 Oct 30 05:46:50.297689 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.297658 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.297831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.297770 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr2kq\" (UniqueName: \"kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.297994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.297863 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.298433 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.298406 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.300711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.300689 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.321775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.321743 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr2kq\" (UniqueName: \"kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq\") pod \"maas-api-revoke-sub-502017e5-cdmj9\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.502377 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.502354 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:50.610470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.610434 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:50.627170 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.627138 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:50.651081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.651046 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:46:50.651258 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.651100 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:46:50.838450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:50.838426 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9"] Oct 30 05:46:50.839660 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:46:50.839636 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd246ee08_db83_42f1_a7ca_e2fefd0c004a.slice/crio-ab3930e8d98cd91a35d122dc97703a4d3bf06900b4307dd01d85e161d94a291e WatchSource:0}: Error finding container ab3930e8d98cd91a35d122dc97703a4d3bf06900b4307dd01d85e161d94a291e: Status 404 returned error can't find the container with id ab3930e8d98cd91a35d122dc97703a4d3bf06900b4307dd01d85e161d94a291e Oct 30 05:46:51.095394 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.095355 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" event={"ID":"d246ee08-db83-42f1-a7ca-e2fefd0c004a","Type":"ContainerStarted","Data":"ab3930e8d98cd91a35d122dc97703a4d3bf06900b4307dd01d85e161d94a291e"} Oct 30 05:46:51.097021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.096991 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" event={"ID":"fe5f6735-b306-4bc1-af35-741411c0b65e","Type":"ContainerStarted","Data":"817f3b4a23c4baef40774109aa4d479f8a93baf33d9f620715f34f2db2b700f9"} Oct 30 05:46:51.097128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.097046 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" event={"ID":"fe5f6735-b306-4bc1-af35-741411c0b65e","Type":"ContainerStarted","Data":"fb4bb8910ebbf817611efa4680e9ae6e289bf4d41c263c5d4a0aa4afec48cf51"} Oct 30 05:46:51.108802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.108778 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:46:51.140536 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.140482 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" podStartSLOduration=2.140469476 podStartE2EDuration="2.140469476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:46:51.139060481 +0000 UTC m=+917.714747535" watchObservedRunningTime="2026-10-30 05:46:51.140469476 +0000 UTC m=+917.716156547" Oct 30 05:46:51.650268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.650183 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:46:51.650268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:51.650233 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:46:52.105148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:52.105116 5565 generic.go:356] "Generic (PLEG): container finished" podID="d246ee08-db83-42f1-a7ca-e2fefd0c004a" containerID="45f730e39b9906676fbaf38ad7d0ad05000818c0ab8e944a3d64a79cbf70e725" exitCode=0 Oct 30 05:46:52.105524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:52.105194 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" event={"ID":"d246ee08-db83-42f1-a7ca-e2fefd0c004a","Type":"ContainerDied","Data":"45f730e39b9906676fbaf38ad7d0ad05000818c0ab8e944a3d64a79cbf70e725"} Oct 30 05:46:52.106839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:52.106816 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe5f6735-b306-4bc1-af35-741411c0b65e" containerID="817f3b4a23c4baef40774109aa4d479f8a93baf33d9f620715f34f2db2b700f9" exitCode=0 Oct 30 05:46:52.106943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:52.106900 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" event={"ID":"fe5f6735-b306-4bc1-af35-741411c0b65e","Type":"ContainerDied","Data":"817f3b4a23c4baef40774109aa4d479f8a93baf33d9f620715f34f2db2b700f9"} Oct 30 05:46:52.650401 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:52.650367 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:46:52.650584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:52.650412 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:46:53.332639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.332613 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:53.335790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.335772 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:53.436856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.436781 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca\") pod \"fe5f6735-b306-4bc1-af35-741411c0b65e\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " Oct 30 05:46:53.436856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.436842 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh\" (UniqueName: \"kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh\") pod \"fe5f6735-b306-4bc1-af35-741411c0b65e\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " Oct 30 05:46:53.437108 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437013 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq\" (UniqueName: \"kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq\") pod \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " Oct 30 05:46:53.437160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437103 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca\") pod \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " Oct 30 05:46:53.437273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437253 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca\") pod \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\" (UID: \"d246ee08-db83-42f1-a7ca-e2fefd0c004a\") " Oct 30 05:46:53.437322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437307 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca\") pod \"fe5f6735-b306-4bc1-af35-741411c0b65e\" (UID: \"fe5f6735-b306-4bc1-af35-741411c0b65e\") " Oct 30 05:46:53.437751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437685 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca" pod "d246ee08-db83-42f1-a7ca-e2fefd0c004a" (UID: "d246ee08-db83-42f1-a7ca-e2fefd0c004a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:53.437751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437725 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca" pod "fe5f6735-b306-4bc1-af35-741411c0b65e" (UID: "fe5f6735-b306-4bc1-af35-741411c0b65e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:46:53.437921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437870 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:53.437921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.437890 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:53.439865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.439840 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca" pod "fe5f6735-b306-4bc1-af35-741411c0b65e" (UID: "fe5f6735-b306-4bc1-af35-741411c0b65e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:53.440664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.440640 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq" pod "d246ee08-db83-42f1-a7ca-e2fefd0c004a" (UID: "d246ee08-db83-42f1-a7ca-e2fefd0c004a"). InnerVolumeSpecName "kube-api-access-kr2kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:53.441916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.441893 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh" pod "fe5f6735-b306-4bc1-af35-741411c0b65e" (UID: "fe5f6735-b306-4bc1-af35-741411c0b65e"). InnerVolumeSpecName "kube-api-access-fp2sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:46:53.443027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.443005 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca" pod "d246ee08-db83-42f1-a7ca-e2fefd0c004a" (UID: "d246ee08-db83-42f1-a7ca-e2fefd0c004a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:46:53.538477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.538449 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fe5f6735-b306-4bc1-af35-741411c0b65e-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:53.538477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.538476 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp2sh\" (UniqueName: \"kubernetes.io/projected/fe5f6735-b306-4bc1-af35-741411c0b65e-kube-api-access-fp2sh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:53.538477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.538486 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr2kq\" (UniqueName: \"kubernetes.io/projected/d246ee08-db83-42f1-a7ca-e2fefd0c004a-kube-api-access-kr2kq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:53.538697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.538495 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d246ee08-db83-42f1-a7ca-e2fefd0c004a-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:46:53.650310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.650276 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 05:46:53.650419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:53.650328 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 05:46:54.120307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.120273 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" Oct 30 05:46:54.120307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.120299 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9" event={"ID":"d246ee08-db83-42f1-a7ca-e2fefd0c004a","Type":"ContainerDied","Data":"ab3930e8d98cd91a35d122dc97703a4d3bf06900b4307dd01d85e161d94a291e"} Oct 30 05:46:54.120595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.120355 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab3930e8d98cd91a35d122dc97703a4d3bf06900b4307dd01d85e161d94a291e" Oct 30 05:46:54.121761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.121743 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" Oct 30 05:46:54.121761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.121743 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd" event={"ID":"fe5f6735-b306-4bc1-af35-741411c0b65e","Type":"ContainerDied","Data":"fb4bb8910ebbf817611efa4680e9ae6e289bf4d41c263c5d4a0aa4afec48cf51"} Oct 30 05:46:54.121950 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.121786 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4bb8910ebbf817611efa4680e9ae6e289bf4d41c263c5d4a0aa4afec48cf51" Oct 30 05:46:54.887730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.887700 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:54.888333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.888114 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:54.889452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:54.889432 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 05:46:55.385292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:46:55.385255 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:47:01.634336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634301 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn"] Oct 30 05:47:01.634856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634840 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe5f6735-b306-4bc1-af35-741411c0b65e" containerName="revoke-keys" Oct 30 05:47:01.634898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634858 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe5f6735-b306-4bc1-af35-741411c0b65e" containerName="revoke-keys" Oct 30 05:47:01.634898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634885 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d246ee08-db83-42f1-a7ca-e2fefd0c004a" containerName="revoke-keys" Oct 30 05:47:01.634898 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634890 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d246ee08-db83-42f1-a7ca-e2fefd0c004a" containerName="revoke-keys" Oct 30 05:47:01.634993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634976 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe5f6735-b306-4bc1-af35-741411c0b65e" containerName="revoke-keys" Oct 30 05:47:01.635025 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.634992 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d246ee08-db83-42f1-a7ca-e2fefd0c004a" containerName="revoke-keys" Oct 30 05:47:01.639734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.639715 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.667047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.667024 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn"] Oct 30 05:47:01.761184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761154 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6jkd\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761190 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761215 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761350 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761382 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761423 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761472 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761576 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.761638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.761607 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.862626 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862594 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6jkd\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.862777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862633 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.862777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862658 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.862777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862707 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.862777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862725 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.862777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862768 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862866 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.862948 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.863062 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.863110 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.863171 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.863296 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863512 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.863493 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.863800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.863768 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.865255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.865238 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.865379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.865361 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.886804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.886742 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.886948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.886929 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6jkd\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:01.952855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:01.952816 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:02.288491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:02.288464 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn"] Oct 30 05:47:02.291311 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:02.291282 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod946e069a_987f_4ede_a2d4_cd376c83d4e7.slice/crio-e19173b0e7dd07041c17a886d35e79098ad76957c8c1ae53500843a793ace729 WatchSource:0}: Error finding container e19173b0e7dd07041c17a886d35e79098ad76957c8c1ae53500843a793ace729: Status 404 returned error can't find the container with id e19173b0e7dd07041c17a886d35e79098ad76957c8c1ae53500843a793ace729 Oct 30 05:47:02.293806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:02.293772 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:02.293915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:02.293836 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:02.293915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:02.293870 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:03.187886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.187843 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" event={"ID":"946e069a-987f-4ede-a2d4-cd376c83d4e7","Type":"ContainerStarted","Data":"95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad"} Oct 30 05:47:03.188267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.187910 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" event={"ID":"946e069a-987f-4ede-a2d4-cd376c83d4e7","Type":"ContainerStarted","Data":"e19173b0e7dd07041c17a886d35e79098ad76957c8c1ae53500843a793ace729"} Oct 30 05:47:03.250429 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.250373 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podStartSLOduration=2.2503561469999998 podStartE2EDuration="2.250356147s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:03.24710116 +0000 UTC m=+929.822788215" watchObservedRunningTime="2026-10-30 05:47:03.250356147 +0000 UTC m=+929.826043202" Oct 30 05:47:03.693146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.693111 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv"] Oct 30 05:47:03.697165 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.697146 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.725983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.725956 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv"] Oct 30 05:47:03.786643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.786604 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.786850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.786657 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.786850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.786782 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.786850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.786831 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.786982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.786866 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.787019 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.786982 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.787060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.787044 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.787135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.787115 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2mzr\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.787186 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.787154 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.888649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888357 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.888649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888439 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.888649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888481 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.888649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888514 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.888649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888601 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888695 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888726 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2mzr\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888753 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888894 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.888991 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.889064 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.889381 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.889755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.889730 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.890008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.889989 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.891697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.891667 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.892098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.892069 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.913138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.913112 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.913138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.913133 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2mzr\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr\") pod \"maas-default-gateway-openshift-default-84587756dc-9b5mv\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:03.977611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.977359 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:03.977791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.977664 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:03.977791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:03.977733 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 05:47:04.011681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.011654 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:04.383841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.383807 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv"] Oct 30 05:47:04.384214 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:04.384163 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1711d7c4_786c_436a_bea7_df643516fdcc.slice/crio-1d269bb8fba9444aa097617a29c2e38204c5aabf2cc1a92fc9a573042ae3336c WatchSource:0}: Error finding container 1d269bb8fba9444aa097617a29c2e38204c5aabf2cc1a92fc9a573042ae3336c: Status 404 returned error can't find the container with id 1d269bb8fba9444aa097617a29c2e38204c5aabf2cc1a92fc9a573042ae3336c Oct 30 05:47:04.387166 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.387125 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:04.387303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.387204 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:04.387303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.387232 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:04.957557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.956884 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:04.957557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:04.956978 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 05:47:05.212613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:05.211976 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" event={"ID":"1711d7c4-786c-436a-bea7-df643516fdcc","Type":"ContainerStarted","Data":"4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58"} Oct 30 05:47:05.212613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:05.212020 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" event={"ID":"1711d7c4-786c-436a-bea7-df643516fdcc","Type":"ContainerStarted","Data":"1d269bb8fba9444aa097617a29c2e38204c5aabf2cc1a92fc9a573042ae3336c"} Oct 30 05:47:05.973553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:05.956912 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:05.973553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:05.956971 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 05:47:06.012282 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:06.012236 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:06.013833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:06.013700 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:06.013833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:06.013756 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:06.959555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:06.956819 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:06.959555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:06.956878 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 05:47:07.013933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:07.013900 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:07.014312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:07.013984 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:07.960553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:07.956888 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:07.960553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:07.956964 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 05:47:08.012869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:08.012836 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:08.013050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:08.012890 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:08.958561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:08.957930 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 05:47:08.958561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:08.957992 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 05:47:09.012495 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:09.012462 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:09.012705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:09.012514 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:09.983099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:09.983058 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:09.983636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:09.983452 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:09.984501 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:09.984473 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 05:47:10.012144 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:10.012104 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:10.012363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:10.012161 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:10.047645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:10.047584 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podStartSLOduration=7.047568524 podStartE2EDuration="7.047568524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:05.272877448 +0000 UTC m=+931.848564503" watchObservedRunningTime="2026-10-30 05:47:10.047568524 +0000 UTC m=+936.623255577" Oct 30 05:47:11.013065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:11.013035 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:11.013509 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:11.013089 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:12.012967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:12.012932 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-9b5mv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 30 05:47:12.013197 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:12.012984 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 30 05:47:13.240303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:13.240184 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:13.243964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:13.243941 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:13.285766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:13.284036 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:47:15.518545 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.518506 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp"] Oct 30 05:47:15.524862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.524838 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.543209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.543174 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp"] Oct 30 05:47:15.552700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.552668 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-64123f\"/\"distinct-w6-64123f-kserve-self-signed-certs\"" Oct 30 05:47:15.638623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.638592 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.638811 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.638722 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.638811 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.638789 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fddb6\" (UniqueName: \"kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.638920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.638886 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.638984 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.639057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.639021 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.739875 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.739838 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.739875 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.739885 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.740149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.740052 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.740241 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.740224 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.740334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.740276 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fddb6\" (UniqueName: \"kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.740549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.740517 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.745774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.741707 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.745774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.742893 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.746969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.746944 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.753157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.753133 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.754198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.754178 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.768799 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.768716 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fddb6\" (UniqueName: \"kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6\") pod \"distinct-w6-64123f-kserve-745f554c55-lqqxp\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:15.839262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:15.839234 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:16.142378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.142346 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp"] Oct 30 05:47:16.142827 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:16.142805 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9982d791_7470_4816_9b2b_7229801b63d7.slice/crio-643d47e5bbde6db2d57004f33fc169f104f51f1bf1915bb76d2b9a3759a78ae6 WatchSource:0}: Error finding container 643d47e5bbde6db2d57004f33fc169f104f51f1bf1915bb76d2b9a3759a78ae6: Status 404 returned error can't find the container with id 643d47e5bbde6db2d57004f33fc169f104f51f1bf1915bb76d2b9a3759a78ae6 Oct 30 05:47:16.360285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.360236 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" event={"ID":"9982d791-7470-4816-9b2b-7229801b63d7","Type":"ContainerStarted","Data":"3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a"} Oct 30 05:47:16.360575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.360493 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" event={"ID":"9982d791-7470-4816-9b2b-7229801b63d7","Type":"ContainerStarted","Data":"643d47e5bbde6db2d57004f33fc169f104f51f1bf1915bb76d2b9a3759a78ae6"} Oct 30 05:47:16.430725 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.430643 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podStartSLOduration=1.430631504 podStartE2EDuration="1.430631504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:16.42957423 +0000 UTC m=+943.005261284" watchObservedRunningTime="2026-10-30 05:47:16.430631504 +0000 UTC m=+943.006318557" Oct 30 05:47:16.829672 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.829640 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m"] Oct 30 05:47:16.833786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.833766 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.839934 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.839912 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:16.956641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956608 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.956823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956663 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.956823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956759 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.956823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956815 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.956964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956915 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.956964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956950 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.957073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.956977 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.957157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.957141 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:16.957214 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:16.957176 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.009975 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.009949 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m"] Oct 30 05:47:17.058144 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058107 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058342 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058289 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058361 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058395 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058459 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058540 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058646 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058584 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058646 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058605 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058646 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058627 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058738 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058804 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058895 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058926 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.058984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.058908 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.060870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.060848 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.061182 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.061163 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.087303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.087244 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.100262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.100231 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.149171 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.149146 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:17.507807 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.507781 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m"] Oct 30 05:47:17.509284 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:17.509258 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44d3ab58_857a_4d45_abdb_5e05d101895c.slice/crio-c9b9dfeefa3ecc8359c8025068a10104e0a2e2993d251d75b73f9c102689a119 WatchSource:0}: Error finding container c9b9dfeefa3ecc8359c8025068a10104e0a2e2993d251d75b73f9c102689a119: Status 404 returned error can't find the container with id c9b9dfeefa3ecc8359c8025068a10104e0a2e2993d251d75b73f9c102689a119 Oct 30 05:47:17.511759 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.511728 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:17.511832 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.511791 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:17.511832 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:17.511826 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:17.535561 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:47:17.535511 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:47:18.385221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:18.385176 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" event={"ID":"44d3ab58-857a-4d45-abdb-5e05d101895c","Type":"ContainerStarted","Data":"86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5"} Oct 30 05:47:18.385723 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:18.385247 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" event={"ID":"44d3ab58-857a-4d45-abdb-5e05d101895c","Type":"ContainerStarted","Data":"c9b9dfeefa3ecc8359c8025068a10104e0a2e2993d251d75b73f9c102689a119"} Oct 30 05:47:18.446475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:18.446423 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podStartSLOduration=2.446410216 podStartE2EDuration="2.446410216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:18.443679453 +0000 UTC m=+945.019366509" watchObservedRunningTime="2026-10-30 05:47:18.446410216 +0000 UTC m=+945.022097350" Oct 30 05:47:19.149821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:19.149786 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:19.150937 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:19.150913 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:47:19.151082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:19.150965 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 05:47:20.149814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:20.149774 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:47:20.150193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:20.149838 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 05:47:21.149561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:21.149513 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:47:21.149747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:21.149615 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 05:47:22.149778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:22.149738 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:47:22.150194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:22.149801 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 05:47:23.150100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:23.150072 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 05:47:23.150512 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:23.150117 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 05:47:24.526445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:24.526412 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:24.526969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:24.526915 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:24.528395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:24.528374 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 05:47:25.435703 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:25.435672 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:47:25.840047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:25.840011 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:25.853207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:25.853183 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:26.459655 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:26.459620 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:47:31.751388 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.751352 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g"] Oct 30 05:47:31.755766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.755746 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.789311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.789279 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g"] Oct 30 05:47:31.876940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.876904 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfgs2\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.876965 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877037 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877070 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877104 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877196 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877213 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877253 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.877373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.877357 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.978920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.978872 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979031 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979059 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979170 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979275 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979346 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979318 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979409 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979367 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfgs2\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979420 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979441 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979485 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979543 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979720 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979687 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.979902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.979879 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.980200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.980181 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.981708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.981686 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:31.982211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:31.982190 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:32.003101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.003066 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:32.003216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.003129 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfgs2\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2\") pod \"e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:32.070354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.070316 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:32.392603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.392575 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g"] Oct 30 05:47:32.393388 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:32.393359 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16768327_985e_4f85_9e4f_85d0eb1c191b.slice/crio-d5c810572dfdcf9dd156bbf0288361329d99a5a015f1f2e88885e6904a311ac5 WatchSource:0}: Error finding container d5c810572dfdcf9dd156bbf0288361329d99a5a015f1f2e88885e6904a311ac5: Status 404 returned error can't find the container with id d5c810572dfdcf9dd156bbf0288361329d99a5a015f1f2e88885e6904a311ac5 Oct 30 05:47:32.395876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.395841 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:32.395964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.395911 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:32.395964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.395946 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:32.484749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.484716 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" event={"ID":"16768327-985e-4f85-9e4f-85d0eb1c191b","Type":"ContainerStarted","Data":"7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe"} Oct 30 05:47:32.484920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.484779 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" event={"ID":"16768327-985e-4f85-9e4f-85d0eb1c191b","Type":"ContainerStarted","Data":"d5c810572dfdcf9dd156bbf0288361329d99a5a015f1f2e88885e6904a311ac5"} Oct 30 05:47:32.536063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:32.536016 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podStartSLOduration=1.5360018800000002 podStartE2EDuration="1.53600188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:32.531203354 +0000 UTC m=+959.106890408" watchObservedRunningTime="2026-10-30 05:47:32.53600188 +0000 UTC m=+959.111688925" Oct 30 05:47:33.075345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:33.075265 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:33.077744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:33.077476 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:33.077744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:33.077553 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 05:47:34.078051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:34.078018 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:34.078622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:34.078588 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 05:47:35.077921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.077884 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:35.078102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.077958 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 05:47:35.780386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.780349 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp"] Oct 30 05:47:35.785974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.785949 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.798655 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.798632 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp"] Oct 30 05:47:35.807015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.806975 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:35.807015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.806978 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:35.807208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.807069 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:47:35.807208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.807105 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:35.826594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.826575 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.826708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.826643 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmkb\" (UniqueName: \"kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.826763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.826743 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.927806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.927775 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.927962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.927917 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmkb\" (UniqueName: \"kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.927962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.927950 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.928559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.928515 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.930381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.930359 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:35.952231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:35.952211 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmkb\" (UniqueName: \"kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb\") pod \"maas-api-revoke-sub-7335ba6e-5vrjp\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:36.078985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:36.078899 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:36.078985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:36.078965 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 05:47:36.103251 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:36.103218 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:36.430965 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:36.430931 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4395bb69_11d9_4aaf_932a_37c233f8861f.slice/crio-55f83be5957e7352252a148902fd0eaa15afdc38b6380ace008e0de9b043ba5d WatchSource:0}: Error finding container 55f83be5957e7352252a148902fd0eaa15afdc38b6380ace008e0de9b043ba5d: Status 404 returned error can't find the container with id 55f83be5957e7352252a148902fd0eaa15afdc38b6380ace008e0de9b043ba5d Oct 30 05:47:36.431278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:36.431252 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp"] Oct 30 05:47:36.513389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:36.513346 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" event={"ID":"4395bb69-11d9-4aaf-932a-37c233f8861f","Type":"ContainerStarted","Data":"55f83be5957e7352252a148902fd0eaa15afdc38b6380ace008e0de9b043ba5d"} Oct 30 05:47:37.077555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.072482 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:37.077555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.072580 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 05:47:37.521549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.521497 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" event={"ID":"4395bb69-11d9-4aaf-932a-37c233f8861f","Type":"ContainerStarted","Data":"c5345a07cfd17e9fee41cce0cbddeb643c511dc62bd7b342e9c1003fc911e7d4"} Oct 30 05:47:37.590735 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.590685 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" podStartSLOduration=2.590669776 podStartE2EDuration="2.590669776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:37.579100739 +0000 UTC m=+964.154787794" watchObservedRunningTime="2026-10-30 05:47:37.590669776 +0000 UTC m=+964.166356829" Oct 30 05:47:37.591752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.591723 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv"] Oct 30 05:47:37.597042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.597017 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.605051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.605024 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv"] Oct 30 05:47:37.621837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.621803 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"openshift-service-ca.crt\"" Oct 30 05:47:37.621837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.621825 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"kube-root-ca.crt\"" Oct 30 05:47:37.622115 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.622029 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"distinct-2-w1-1d8c2f-kserve-self-signed-certs\"" Oct 30 05:47:37.622165 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.622128 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-1d8c2f\"/\"default-dockercfg-lkcsn\"" Oct 30 05:47:37.650082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.650049 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.650235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.650096 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.650235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.650211 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk7x4\" (UniqueName: \"kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.650457 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.650435 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.650546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.650487 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.650670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.650640 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752245 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752202 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk7x4\" (UniqueName: \"kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752355 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752380 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752428 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752455 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752495 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.752936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.752878 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.753643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.753619 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.781612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.769277 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.795554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.784477 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.795554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.793656 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk7x4\" (UniqueName: \"kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.796129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.796094 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm\") pod \"distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:37.988405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:37.988366 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:38.085565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.084075 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 05:47:38.085565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.084186 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 05:47:38.341058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.341020 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv"] Oct 30 05:47:38.341598 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:38.341567 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3482a4c5_784b_4f8c_9e34_83429255b0b0.slice/crio-5644ad36d55ac8bc3fa07e5956b53e99fe2c3d12eb090e66ca803f85106d81f0 WatchSource:0}: Error finding container 5644ad36d55ac8bc3fa07e5956b53e99fe2c3d12eb090e66ca803f85106d81f0: Status 404 returned error can't find the container with id 5644ad36d55ac8bc3fa07e5956b53e99fe2c3d12eb090e66ca803f85106d81f0 Oct 30 05:47:38.530861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.530821 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" event={"ID":"3482a4c5-784b-4f8c-9e34-83429255b0b0","Type":"ContainerStarted","Data":"d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7"} Oct 30 05:47:38.531360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.530869 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" event={"ID":"3482a4c5-784b-4f8c-9e34-83429255b0b0","Type":"ContainerStarted","Data":"5644ad36d55ac8bc3fa07e5956b53e99fe2c3d12eb090e66ca803f85106d81f0"} Oct 30 05:47:38.532879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.532845 5565 generic.go:356] "Generic (PLEG): container finished" podID="4395bb69-11d9-4aaf-932a-37c233f8861f" containerID="c5345a07cfd17e9fee41cce0cbddeb643c511dc62bd7b342e9c1003fc911e7d4" exitCode=0 Oct 30 05:47:38.533000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.532918 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" event={"ID":"4395bb69-11d9-4aaf-932a-37c233f8861f","Type":"ContainerDied","Data":"c5345a07cfd17e9fee41cce0cbddeb643c511dc62bd7b342e9c1003fc911e7d4"} Oct 30 05:47:38.582948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.582891 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podStartSLOduration=1.582874684 podStartE2EDuration="1.582874684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:38.581602016 +0000 UTC m=+965.157289139" watchObservedRunningTime="2026-10-30 05:47:38.582874684 +0000 UTC m=+965.158561738" Oct 30 05:47:38.989428 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:38.989386 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:39.401056 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.401023 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:39.401403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.401377 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:39.402386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.402361 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:47:39.701910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.701886 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:39.811005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.810976 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca\") pod \"4395bb69-11d9-4aaf-932a-37c233f8861f\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " Oct 30 05:47:39.811162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.811017 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca\") pod \"4395bb69-11d9-4aaf-932a-37c233f8861f\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " Oct 30 05:47:39.811162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.811035 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb\" (UniqueName: \"kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb\") pod \"4395bb69-11d9-4aaf-932a-37c233f8861f\" (UID: \"4395bb69-11d9-4aaf-932a-37c233f8861f\") " Oct 30 05:47:39.811413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.811378 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca" pod "4395bb69-11d9-4aaf-932a-37c233f8861f" (UID: "4395bb69-11d9-4aaf-932a-37c233f8861f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:39.813006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.812984 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca" pod "4395bb69-11d9-4aaf-932a-37c233f8861f" (UID: "4395bb69-11d9-4aaf-932a-37c233f8861f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:39.813126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.813108 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb" pod "4395bb69-11d9-4aaf-932a-37c233f8861f" (UID: "4395bb69-11d9-4aaf-932a-37c233f8861f"). InnerVolumeSpecName "kube-api-access-pgmkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:39.912360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.912335 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:47:39.912360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.912357 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgmkb\" (UniqueName: \"kubernetes.io/projected/4395bb69-11d9-4aaf-932a-37c233f8861f-kube-api-access-pgmkb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:47:39.912497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:39.912367 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4395bb69-11d9-4aaf-932a-37c233f8861f-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:47:40.557455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:40.557427 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" Oct 30 05:47:40.557653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:40.557463 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp" event={"ID":"4395bb69-11d9-4aaf-932a-37c233f8861f","Type":"ContainerDied","Data":"55f83be5957e7352252a148902fd0eaa15afdc38b6380ace008e0de9b043ba5d"} Oct 30 05:47:40.557653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:40.557565 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55f83be5957e7352252a148902fd0eaa15afdc38b6380ace008e0de9b043ba5d" Oct 30 05:47:46.614080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.614047 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh"] Oct 30 05:47:46.614652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.614634 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4395bb69-11d9-4aaf-932a-37c233f8861f" containerName="revoke-keys" Oct 30 05:47:46.614652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.614651 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4395bb69-11d9-4aaf-932a-37c233f8861f" containerName="revoke-keys" Oct 30 05:47:46.614801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.614752 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4395bb69-11d9-4aaf-932a-37c233f8861f" containerName="revoke-keys" Oct 30 05:47:46.618465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.618445 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.664854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.664587 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh"] Oct 30 05:47:46.704282 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704229 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704288 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704337 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704392 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w8lt\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704446 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704565 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704659 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704733 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.704928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.704770 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806320 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w8lt\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806401 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806544 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806580 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806624 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806678 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806782 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806813 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.806883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.806867 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.807501 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.807475 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.807590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.807493 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.807841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.807794 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.807966 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.807913 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.808510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.808484 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.810276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.810254 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.810638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.810614 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.831798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.831763 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w8lt\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.831934 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.831869 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:46.948067 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:46.947617 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:47.557128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:47.557015 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh"] Oct 30 05:47:47.638308 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:47:47.638151 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:47:47.638308 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:47.638270 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd67151c_42b7_4c88_8f69_a90be47788c4.slice/crio-44658fd93b065e5acd5843748d67f94d8482117995230d9a5d1cd589f502715d WatchSource:0}: Error finding container 44658fd93b065e5acd5843748d67f94d8482117995230d9a5d1cd589f502715d: Status 404 returned error can't find the container with id 44658fd93b065e5acd5843748d67f94d8482117995230d9a5d1cd589f502715d Oct 30 05:47:47.722555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:47.711004 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:47.722555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:47.711170 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:47.722555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:47.711256 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:47:47.989474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:47.989445 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:48.004007 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.003980 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:48.787568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.787521 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" event={"ID":"bd67151c-42b7-4c88-8f69-a90be47788c4","Type":"ContainerStarted","Data":"4c1d8d6b4a577d957208d4b592b408a2924f6243b046f824f5a6be7e16fa00e4"} Oct 30 05:47:48.788053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.787590 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" event={"ID":"bd67151c-42b7-4c88-8f69-a90be47788c4","Type":"ContainerStarted","Data":"44658fd93b065e5acd5843748d67f94d8482117995230d9a5d1cd589f502715d"} Oct 30 05:47:48.798977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.798948 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:47:48.849396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.849350 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podStartSLOduration=2.849336871 podStartE2EDuration="2.849336871s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:48.84820429 +0000 UTC m=+975.423891344" watchObservedRunningTime="2026-10-30 05:47:48.849336871 +0000 UTC m=+975.425023925" Oct 30 05:47:48.948591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.948558 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:48.949985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.949952 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:48.950127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:48.950016 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 05:47:49.949077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:49.949047 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:49.949462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:49.949103 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 05:47:50.948399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:50.948368 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:50.948590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:50.948417 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 05:47:51.948649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:51.948614 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:51.949046 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:51.948663 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 05:47:52.948952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:52.948922 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:52.949399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:52.948979 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 05:47:53.428267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.428225 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz"] Oct 30 05:47:53.434377 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.434340 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.482420 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.482382 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz"] Oct 30 05:47:53.482873 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.482838 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-fffaa8\"/\"distinct-2-w3-fffaa8-kserve-self-signed-certs\"" Oct 30 05:47:53.537085 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.537053 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h"] Oct 30 05:47:53.541136 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.541118 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.541742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.541717 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vpxkn\" (UniqueName: \"kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.541843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.541756 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.541887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.541837 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.541887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.541864 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.541965 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.541909 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.542080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.542057 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.578179 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.578151 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:47:53.579604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.578785 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:47:53.579765 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.578857 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:47:53.579868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.578943 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:47:53.591041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.591018 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h"] Oct 30 05:47:53.642832 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.642802 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.642993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.642928 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vpxkn\" (UniqueName: \"kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.642993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.642945 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.642993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.642973 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.643156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.642998 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.643156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643062 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.643296 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643267 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.643296 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643283 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mbf\" (UniqueName: \"kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.643407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643347 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.643407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643350 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.643517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643443 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.643655 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.643621 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.645412 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.645388 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.646194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.646173 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.693377 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.693297 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vpxkn\" (UniqueName: \"kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn\") pod \"distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.744888 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.744855 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mbf\" (UniqueName: \"kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.745069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.744934 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.745177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.745155 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.746089 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.746064 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.747673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.747648 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.760837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.760789 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:53.793558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.788593 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mbf\" (UniqueName: \"kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf\") pod \"maas-api-revoke-sub-8efe04be-sh49h\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.883921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.883881 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:53.949251 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.949116 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 30 05:47:53.949251 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:53.949186 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 30 05:47:54.105467 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:54.105425 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod29a9f246_2ac1_4eb4_a230_8d028068b3dd.slice/crio-a514d820fe36aa4250ca1de34e58b9353471ef7dccd30e425488e57b2f6a8904 WatchSource:0}: Error finding container a514d820fe36aa4250ca1de34e58b9353471ef7dccd30e425488e57b2f6a8904: Status 404 returned error can't find the container with id a514d820fe36aa4250ca1de34e58b9353471ef7dccd30e425488e57b2f6a8904 Oct 30 05:47:54.105624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.105600 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz"] Oct 30 05:47:54.232470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.232393 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h"] Oct 30 05:47:54.235624 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:47:54.235592 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53f57850_b05c_476c_a1c8_4737a072ed55.slice/crio-d4864dbe08ce9a0bf42fe13555194d47ba8ca1ff27932c7a06227557cff81450 WatchSource:0}: Error finding container d4864dbe08ce9a0bf42fe13555194d47ba8ca1ff27932c7a06227557cff81450: Status 404 returned error can't find the container with id d4864dbe08ce9a0bf42fe13555194d47ba8ca1ff27932c7a06227557cff81450 Oct 30 05:47:54.831431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.831319 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" event={"ID":"53f57850-b05c-476c-a1c8-4737a072ed55","Type":"ContainerStarted","Data":"fd3b5580e1195d8dff3fbe6c57cb82ee9d0af3461db197fbf3e07a0ac78544e8"} Oct 30 05:47:54.831431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.831378 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" event={"ID":"53f57850-b05c-476c-a1c8-4737a072ed55","Type":"ContainerStarted","Data":"d4864dbe08ce9a0bf42fe13555194d47ba8ca1ff27932c7a06227557cff81450"} Oct 30 05:47:54.833306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.833280 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" event={"ID":"29a9f246-2ac1-4eb4-a230-8d028068b3dd","Type":"ContainerStarted","Data":"12f1d15106a4c3ca7665857e14f5894e7384012b31f6234d27b886798118ade1"} Oct 30 05:47:54.833434 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.833315 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" event={"ID":"29a9f246-2ac1-4eb4-a230-8d028068b3dd","Type":"ContainerStarted","Data":"a514d820fe36aa4250ca1de34e58b9353471ef7dccd30e425488e57b2f6a8904"} Oct 30 05:47:54.917258 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.917208 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" podStartSLOduration=1.917195559 podStartE2EDuration="1.917195559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:54.916172422 +0000 UTC m=+981.491859484" watchObservedRunningTime="2026-10-30 05:47:54.917195559 +0000 UTC m=+981.492882612" Oct 30 05:47:54.962430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:54.962374 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podStartSLOduration=1.962237214 podStartE2EDuration="1.962237214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:47:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:47:54.95894193 +0000 UTC m=+981.534628984" watchObservedRunningTime="2026-10-30 05:47:54.962237214 +0000 UTC m=+981.537924269" Oct 30 05:47:55.195851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:55.195825 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:55.196354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:55.196319 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:55.197247 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:55.197222 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 05:47:55.764868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:55.764834 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:47:55.843412 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:55.843377 5565 generic.go:356] "Generic (PLEG): container finished" podID="53f57850-b05c-476c-a1c8-4737a072ed55" containerID="fd3b5580e1195d8dff3fbe6c57cb82ee9d0af3461db197fbf3e07a0ac78544e8" exitCode=0 Oct 30 05:47:55.843625 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:55.843474 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" event={"ID":"53f57850-b05c-476c-a1c8-4737a072ed55","Type":"ContainerDied","Data":"fd3b5580e1195d8dff3fbe6c57cb82ee9d0af3461db197fbf3e07a0ac78544e8"} Oct 30 05:47:57.016760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.016736 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:57.093964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.093930 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca\") pod \"53f57850-b05c-476c-a1c8-4737a072ed55\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " Oct 30 05:47:57.094157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.094140 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca\") pod \"53f57850-b05c-476c-a1c8-4737a072ed55\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " Oct 30 05:47:57.094274 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.094250 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf\" (UniqueName: \"kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf\") pod \"53f57850-b05c-476c-a1c8-4737a072ed55\" (UID: \"53f57850-b05c-476c-a1c8-4737a072ed55\") " Oct 30 05:47:57.094559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.094516 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca" pod "53f57850-b05c-476c-a1c8-4737a072ed55" (UID: "53f57850-b05c-476c-a1c8-4737a072ed55"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:47:57.094763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.094744 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:47:57.096085 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.096060 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca" pod "53f57850-b05c-476c-a1c8-4737a072ed55" (UID: "53f57850-b05c-476c-a1c8-4737a072ed55"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:47:57.096315 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.096296 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf" pod "53f57850-b05c-476c-a1c8-4737a072ed55" (UID: "53f57850-b05c-476c-a1c8-4737a072ed55"). InnerVolumeSpecName "kube-api-access-j9mbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:47:57.195462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.195431 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9mbf\" (UniqueName: \"kubernetes.io/projected/53f57850-b05c-476c-a1c8-4737a072ed55-kube-api-access-j9mbf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:47:57.195462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.195457 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53f57850-b05c-476c-a1c8-4737a072ed55-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:47:57.861312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.861228 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" Oct 30 05:47:57.861312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.861277 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h" event={"ID":"53f57850-b05c-476c-a1c8-4737a072ed55","Type":"ContainerDied","Data":"d4864dbe08ce9a0bf42fe13555194d47ba8ca1ff27932c7a06227557cff81450"} Oct 30 05:47:57.861479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:47:57.861322 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4864dbe08ce9a0bf42fe13555194d47ba8ca1ff27932c7a06227557cff81450" Oct 30 05:48:03.765389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:03.765346 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:48:03.778321 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:03.778289 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:48:03.914327 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:03.914293 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:48:05.585159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:05.585126 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:48:09.463075 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.463040 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq"] Oct 30 05:48:09.463815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.463744 5565 cpu_manager.go:414] "containerMap: removing container" podUID="53f57850-b05c-476c-a1c8-4737a072ed55" containerName="revoke-keys" Oct 30 05:48:09.463815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.463760 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53f57850-b05c-476c-a1c8-4737a072ed55" containerName="revoke-keys" Oct 30 05:48:09.463937 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.463861 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="53f57850-b05c-476c-a1c8-4737a072ed55" containerName="revoke-keys" Oct 30 05:48:09.468701 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.468682 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.506699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.506664 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:09.506699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.506694 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:09.506897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.506752 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:09.506897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.506783 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:48:09.507286 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.507258 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq"] Oct 30 05:48:09.536913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.536877 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.537153 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.537043 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.537153 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.537110 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mt6h\" (UniqueName: \"kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.638057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.638023 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.638253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.638076 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mt6h\" (UniqueName: \"kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.638253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.638188 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.638824 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.638800 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.640454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.640428 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.661785 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.661752 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mt6h\" (UniqueName: \"kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h\") pod \"maas-api-revoke-sub-b37365e9-tnlgq\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:09.783402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:09.783350 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:10.136665 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:10.136623 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f79bf49_efb1_4ef2_bd25_a1d8eb504483.slice/crio-8a32d58989a72f40e9866d3b2e04cff6c95c2b7b694933b35ecfc561b02976f7 WatchSource:0}: Error finding container 8a32d58989a72f40e9866d3b2e04cff6c95c2b7b694933b35ecfc561b02976f7: Status 404 returned error can't find the container with id 8a32d58989a72f40e9866d3b2e04cff6c95c2b7b694933b35ecfc561b02976f7 Oct 30 05:48:10.137090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:10.137065 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq"] Oct 30 05:48:11.004774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:11.004748 5565 generic.go:356] "Generic (PLEG): container finished" podID="6f79bf49-efb1-4ef2-bd25-a1d8eb504483" containerID="22f212fc610160adc9d3f06ad701154ccc84c5edf15bd0cb2217b7586f8905fc" exitCode=0 Oct 30 05:48:11.005099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:11.004827 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" event={"ID":"6f79bf49-efb1-4ef2-bd25-a1d8eb504483","Type":"ContainerDied","Data":"22f212fc610160adc9d3f06ad701154ccc84c5edf15bd0cb2217b7586f8905fc"} Oct 30 05:48:11.005099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:11.004880 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" event={"ID":"6f79bf49-efb1-4ef2-bd25-a1d8eb504483","Type":"ContainerStarted","Data":"8a32d58989a72f40e9866d3b2e04cff6c95c2b7b694933b35ecfc561b02976f7"} Oct 30 05:48:12.156863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.156840 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:12.168066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.168047 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h\" (UniqueName: \"kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h\") pod \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " Oct 30 05:48:12.168173 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.168160 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca\") pod \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " Oct 30 05:48:12.168260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.168243 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca\") pod \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\" (UID: \"6f79bf49-efb1-4ef2-bd25-a1d8eb504483\") " Oct 30 05:48:12.168601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.168565 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca" pod "6f79bf49-efb1-4ef2-bd25-a1d8eb504483" (UID: "6f79bf49-efb1-4ef2-bd25-a1d8eb504483"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:12.168711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.168682 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:12.170138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.170112 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h" pod "6f79bf49-efb1-4ef2-bd25-a1d8eb504483" (UID: "6f79bf49-efb1-4ef2-bd25-a1d8eb504483"). InnerVolumeSpecName "kube-api-access-5mt6h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:12.170231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.170144 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca" pod "6f79bf49-efb1-4ef2-bd25-a1d8eb504483" (UID: "6f79bf49-efb1-4ef2-bd25-a1d8eb504483"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:12.254688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.254654 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx"] Oct 30 05:48:12.255406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.255384 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6f79bf49-efb1-4ef2-bd25-a1d8eb504483" containerName="revoke-keys" Oct 30 05:48:12.255406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.255408 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f79bf49-efb1-4ef2-bd25-a1d8eb504483" containerName="revoke-keys" Oct 30 05:48:12.255608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.255571 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f79bf49-efb1-4ef2-bd25-a1d8eb504483" containerName="revoke-keys" Oct 30 05:48:12.259298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.259276 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.269587 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdw8g\" (UniqueName: \"kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.269732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.269641 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.269809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.269764 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.269946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.269927 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mt6h\" (UniqueName: \"kubernetes.io/projected/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-kube-api-access-5mt6h\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:12.270026 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.269948 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f79bf49-efb1-4ef2-bd25-a1d8eb504483-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:12.280224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.280200 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx"] Oct 30 05:48:12.371030 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.370996 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.371198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.371043 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.371198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.371168 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdw8g\" (UniqueName: \"kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.371813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.371784 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.373325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.373305 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.394914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.394890 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdw8g\" (UniqueName: \"kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g\") pod \"maas-api-revoke-sub-d8ae4dae-7nkpx\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.575406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.575334 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:12.881236 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:12.881205 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx"] Oct 30 05:48:12.882645 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:12.882618 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod746fcd6a_7ea2_4caa_858b_b6c39b208d41.slice/crio-72a2d9d04d6c0a5cdb3e6792399fd18640228478e08c18a417df9c0900ca7cd9 WatchSource:0}: Error finding container 72a2d9d04d6c0a5cdb3e6792399fd18640228478e08c18a417df9c0900ca7cd9: Status 404 returned error can't find the container with id 72a2d9d04d6c0a5cdb3e6792399fd18640228478e08c18a417df9c0900ca7cd9 Oct 30 05:48:13.020301 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:13.020264 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" event={"ID":"746fcd6a-7ea2-4caa-858b-b6c39b208d41","Type":"ContainerStarted","Data":"72a2d9d04d6c0a5cdb3e6792399fd18640228478e08c18a417df9c0900ca7cd9"} Oct 30 05:48:13.021637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:13.021614 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" Oct 30 05:48:13.021637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:13.021623 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq" event={"ID":"6f79bf49-efb1-4ef2-bd25-a1d8eb504483","Type":"ContainerDied","Data":"8a32d58989a72f40e9866d3b2e04cff6c95c2b7b694933b35ecfc561b02976f7"} Oct 30 05:48:13.021801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:13.021669 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a32d58989a72f40e9866d3b2e04cff6c95c2b7b694933b35ecfc561b02976f7" Oct 30 05:48:14.029927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:14.029845 5565 generic.go:356] "Generic (PLEG): container finished" podID="746fcd6a-7ea2-4caa-858b-b6c39b208d41" containerID="53f279b1b80591e992e98a8347bbf0aff699edf800ee73a61c8868fffbc850b1" exitCode=0 Oct 30 05:48:14.030292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:14.029933 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" event={"ID":"746fcd6a-7ea2-4caa-858b-b6c39b208d41","Type":"ContainerDied","Data":"53f279b1b80591e992e98a8347bbf0aff699edf800ee73a61c8868fffbc850b1"} Oct 30 05:48:15.196851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.196822 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:15.312894 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.312812 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca\") pod \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " Oct 30 05:48:15.312894 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.312866 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca\") pod \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " Oct 30 05:48:15.313132 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.312969 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g\" (UniqueName: \"kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g\") pod \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\" (UID: \"746fcd6a-7ea2-4caa-858b-b6c39b208d41\") " Oct 30 05:48:15.313247 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.313223 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca" pod "746fcd6a-7ea2-4caa-858b-b6c39b208d41" (UID: "746fcd6a-7ea2-4caa-858b-b6c39b208d41"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:15.313470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.313447 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:15.315042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.315019 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca" pod "746fcd6a-7ea2-4caa-858b-b6c39b208d41" (UID: "746fcd6a-7ea2-4caa-858b-b6c39b208d41"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:15.315042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.315028 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g" pod "746fcd6a-7ea2-4caa-858b-b6c39b208d41" (UID: "746fcd6a-7ea2-4caa-858b-b6c39b208d41"). InnerVolumeSpecName "kube-api-access-jdw8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:15.414807 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.414780 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/746fcd6a-7ea2-4caa-858b-b6c39b208d41-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:15.414807 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:15.414805 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdw8g\" (UniqueName: \"kubernetes.io/projected/746fcd6a-7ea2-4caa-858b-b6c39b208d41-kube-api-access-jdw8g\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:16.044861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:16.044832 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" Oct 30 05:48:16.044861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:16.044858 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx" event={"ID":"746fcd6a-7ea2-4caa-858b-b6c39b208d41","Type":"ContainerDied","Data":"72a2d9d04d6c0a5cdb3e6792399fd18640228478e08c18a417df9c0900ca7cd9"} Oct 30 05:48:16.045082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:16.044902 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72a2d9d04d6c0a5cdb3e6792399fd18640228478e08c18a417df9c0900ca7cd9" Oct 30 05:48:17.532687 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:48:17.532647 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:48:21.209552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.208610 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:48:21.209552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.209069 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:48:21.221550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.220600 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:21.349556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.342594 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:48:21.451552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.450615 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:48:21.503688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.503644 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:21.607705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.607660 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:48:21.611647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.611560 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:21.788728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.788621 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:21.910675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:21.910622 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:22.270611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:22.269655 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:22.270611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:22.270366 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:48:22.299257 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:22.299224 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:48:23.089278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.089240 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh"] Oct 30 05:48:23.089846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.089830 5565 cpu_manager.go:414] "containerMap: removing container" podUID="746fcd6a-7ea2-4caa-858b-b6c39b208d41" containerName="revoke-keys" Oct 30 05:48:23.089919 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.089847 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="746fcd6a-7ea2-4caa-858b-b6c39b208d41" containerName="revoke-keys" Oct 30 05:48:23.089969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.089960 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="746fcd6a-7ea2-4caa-858b-b6c39b208d41" containerName="revoke-keys" Oct 30 05:48:23.093461 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.093445 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.111868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.111842 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:23.115954 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.115927 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh"] Oct 30 05:48:23.130168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.124652 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:23.130168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.124652 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:23.130168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.124705 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:48:23.174710 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.174682 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.174960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.174940 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.175041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.175002 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rm4r\" (UniqueName: \"kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.276222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.276191 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.276751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.276239 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rm4r\" (UniqueName: \"kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.276751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.276307 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.277011 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.276990 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.278729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.278703 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.301476 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.301447 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rm4r\" (UniqueName: \"kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r\") pod \"maas-api-revoke-sub-a7f6db91-l5dqh\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.371941 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.371869 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2"] Oct 30 05:48:23.376161 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.376143 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.399904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.399873 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2"] Oct 30 05:48:23.408277 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.408245 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:23.478713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.478663 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.479255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.478950 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x25jr\" (UniqueName: \"kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.479255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.479128 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.580314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.580283 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x25jr\" (UniqueName: \"kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.580488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.580390 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.580488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.580420 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.581066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.581031 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.582832 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.582810 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.604473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.604447 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x25jr\" (UniqueName: \"kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr\") pod \"maas-api-revoke-sub-32b3f894-f8kt2\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.690369 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.690298 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:23.722926 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:23.722884 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb1c3d8_569f_4a14_89da_5cd9bbef3a79.slice/crio-a3f7ce94db2707ca03dea9c8f0f6c051989e5dd42a24e2bdab7b4ed812457d3a WatchSource:0}: Error finding container a3f7ce94db2707ca03dea9c8f0f6c051989e5dd42a24e2bdab7b4ed812457d3a: Status 404 returned error can't find the container with id a3f7ce94db2707ca03dea9c8f0f6c051989e5dd42a24e2bdab7b4ed812457d3a Oct 30 05:48:23.723032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:23.723010 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh"] Oct 30 05:48:24.007760 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:24.007728 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99f98609_5305_4aad_86b0_aff342723ea4.slice/crio-38adfbaac2356dcdd689d2fddb6cebf0bbbd789897586d201d121998c4460051 WatchSource:0}: Error finding container 38adfbaac2356dcdd689d2fddb6cebf0bbbd789897586d201d121998c4460051: Status 404 returned error can't find the container with id 38adfbaac2356dcdd689d2fddb6cebf0bbbd789897586d201d121998c4460051 Oct 30 05:48:24.008432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:24.008403 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2"] Oct 30 05:48:24.263780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:24.263740 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" event={"ID":"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79","Type":"ContainerStarted","Data":"7eb52a08fbe884f5b4513ff3d7d7c251126096d63f8a5e64fe4fa856d3b9b2b0"} Oct 30 05:48:24.264023 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:24.263808 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" event={"ID":"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79","Type":"ContainerStarted","Data":"a3f7ce94db2707ca03dea9c8f0f6c051989e5dd42a24e2bdab7b4ed812457d3a"} Oct 30 05:48:24.265149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:24.265126 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" event={"ID":"99f98609-5305-4aad-86b0-aff342723ea4","Type":"ContainerStarted","Data":"38adfbaac2356dcdd689d2fddb6cebf0bbbd789897586d201d121998c4460051"} Oct 30 05:48:24.311556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:24.309417 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" podStartSLOduration=1.3093971930000001 podStartE2EDuration="1.309397193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:24.305905588 +0000 UTC m=+1010.881592663" watchObservedRunningTime="2026-10-30 05:48:24.309397193 +0000 UTC m=+1010.885084247" Oct 30 05:48:25.273293 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:25.273256 5565 generic.go:356] "Generic (PLEG): container finished" podID="0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" containerID="7eb52a08fbe884f5b4513ff3d7d7c251126096d63f8a5e64fe4fa856d3b9b2b0" exitCode=0 Oct 30 05:48:25.273469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:25.273334 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" event={"ID":"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79","Type":"ContainerDied","Data":"7eb52a08fbe884f5b4513ff3d7d7c251126096d63f8a5e64fe4fa856d3b9b2b0"} Oct 30 05:48:25.283302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:25.283268 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" event={"ID":"99f98609-5305-4aad-86b0-aff342723ea4","Type":"ContainerDied","Data":"cacc40b5a94bb554a61951f8eb84b3fe54ee47fe8ce3b4dfc7ffb298d2d5fdfc"} Oct 30 05:48:25.283445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:25.283128 5565 generic.go:356] "Generic (PLEG): container finished" podID="99f98609-5305-4aad-86b0-aff342723ea4" containerID="cacc40b5a94bb554a61951f8eb84b3fe54ee47fe8ce3b4dfc7ffb298d2d5fdfc" exitCode=0 Oct 30 05:48:26.443797 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.443762 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:26.477405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.477382 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:26.620080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620001 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr\" (UniqueName: \"kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr\") pod \"99f98609-5305-4aad-86b0-aff342723ea4\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " Oct 30 05:48:26.620080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620047 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca\") pod \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " Oct 30 05:48:26.620295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620089 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca\") pod \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " Oct 30 05:48:26.620295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620111 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca\") pod \"99f98609-5305-4aad-86b0-aff342723ea4\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " Oct 30 05:48:26.620403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620296 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r\" (UniqueName: \"kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r\") pod \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\" (UID: \"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79\") " Oct 30 05:48:26.620403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620356 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca\") pod \"99f98609-5305-4aad-86b0-aff342723ea4\" (UID: \"99f98609-5305-4aad-86b0-aff342723ea4\") " Oct 30 05:48:26.620518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620487 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca" pod "0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" (UID: "0fb1c3d8-569f-4a14-89da-5cd9bbef3a79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:26.620721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620700 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca" pod "99f98609-5305-4aad-86b0-aff342723ea4" (UID: "99f98609-5305-4aad-86b0-aff342723ea4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:26.621000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.620984 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:26.621058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.621003 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:26.622368 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.622345 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca" pod "99f98609-5305-4aad-86b0-aff342723ea4" (UID: "99f98609-5305-4aad-86b0-aff342723ea4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:26.622782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.622759 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r" pod "0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" (UID: "0fb1c3d8-569f-4a14-89da-5cd9bbef3a79"). InnerVolumeSpecName "kube-api-access-9rm4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:26.622782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.622773 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca" pod "0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" (UID: "0fb1c3d8-569f-4a14-89da-5cd9bbef3a79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:26.622906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.622780 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr" pod "99f98609-5305-4aad-86b0-aff342723ea4" (UID: "99f98609-5305-4aad-86b0-aff342723ea4"). InnerVolumeSpecName "kube-api-access-x25jr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:26.723149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.723113 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rm4r\" (UniqueName: \"kubernetes.io/projected/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-kube-api-access-9rm4r\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:26.723149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.723149 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x25jr\" (UniqueName: \"kubernetes.io/projected/99f98609-5305-4aad-86b0-aff342723ea4-kube-api-access-x25jr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:26.723357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.723166 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:26.723357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:26.723179 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/99f98609-5305-4aad-86b0-aff342723ea4-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:27.297945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.297918 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" Oct 30 05:48:27.298129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.297954 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2" event={"ID":"99f98609-5305-4aad-86b0-aff342723ea4","Type":"ContainerDied","Data":"38adfbaac2356dcdd689d2fddb6cebf0bbbd789897586d201d121998c4460051"} Oct 30 05:48:27.298129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.298000 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38adfbaac2356dcdd689d2fddb6cebf0bbbd789897586d201d121998c4460051" Oct 30 05:48:27.299613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.299587 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" event={"ID":"0fb1c3d8-569f-4a14-89da-5cd9bbef3a79","Type":"ContainerDied","Data":"a3f7ce94db2707ca03dea9c8f0f6c051989e5dd42a24e2bdab7b4ed812457d3a"} Oct 30 05:48:27.299613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.299610 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh" Oct 30 05:48:27.299780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.299629 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3f7ce94db2707ca03dea9c8f0f6c051989e5dd42a24e2bdab7b4ed812457d3a" Oct 30 05:48:27.841365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.841328 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr"] Oct 30 05:48:27.842114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.842091 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" containerName="revoke-keys" Oct 30 05:48:27.842230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.842115 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" containerName="revoke-keys" Oct 30 05:48:27.842230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.842150 5565 cpu_manager.go:414] "containerMap: removing container" podUID="99f98609-5305-4aad-86b0-aff342723ea4" containerName="revoke-keys" Oct 30 05:48:27.842230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.842159 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99f98609-5305-4aad-86b0-aff342723ea4" containerName="revoke-keys" Oct 30 05:48:27.842374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.842332 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" containerName="revoke-keys" Oct 30 05:48:27.842374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.842352 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="99f98609-5305-4aad-86b0-aff342723ea4" containerName="revoke-keys" Oct 30 05:48:27.846253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.846231 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:27.873881 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.873858 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:27.874244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.874223 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:48:27.874244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.874226 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:27.874408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.874248 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:27.891010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.890980 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr"] Oct 30 05:48:27.938734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.938706 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdkzb\" (UniqueName: \"kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:27.938902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.938782 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:27.938902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:27.938889 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.040147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.040115 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdkzb\" (UniqueName: \"kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.040316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.040275 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.040405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.040388 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.041042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.041017 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.042506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.042490 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.065325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.065295 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdkzb\" (UniqueName: \"kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb\") pod \"maas-api-revoke-sub-417adf23-fmpjr\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.160223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.160149 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:28.490325 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:28.490291 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd7c9a2c_409e_4dcd_a9a9_23895bf4f0cf.slice/crio-027c5da16401b5a03c4ed6b7f4081824ad4b2865c23e0f182584dfa9c557dc96 WatchSource:0}: Error finding container 027c5da16401b5a03c4ed6b7f4081824ad4b2865c23e0f182584dfa9c557dc96: Status 404 returned error can't find the container with id 027c5da16401b5a03c4ed6b7f4081824ad4b2865c23e0f182584dfa9c557dc96 Oct 30 05:48:28.491368 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:28.491342 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr"] Oct 30 05:48:29.318866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:29.318835 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" event={"ID":"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf","Type":"ContainerStarted","Data":"6b3507cecfaa7c8f481abae0599f79409bf81b6b203328f765039c27135f6faf"} Oct 30 05:48:29.319233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:29.318888 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" event={"ID":"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf","Type":"ContainerStarted","Data":"027c5da16401b5a03c4ed6b7f4081824ad4b2865c23e0f182584dfa9c557dc96"} Oct 30 05:48:29.372201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:29.372154 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" podStartSLOduration=2.372140681 podStartE2EDuration="2.372140681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:29.36878582 +0000 UTC m=+1015.944472853" watchObservedRunningTime="2026-10-30 05:48:29.372140681 +0000 UTC m=+1015.947827734" Oct 30 05:48:30.327602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.327570 5565 generic.go:356] "Generic (PLEG): container finished" podID="bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" containerID="6b3507cecfaa7c8f481abae0599f79409bf81b6b203328f765039c27135f6faf" exitCode=0 Oct 30 05:48:30.327984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.327656 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" event={"ID":"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf","Type":"ContainerDied","Data":"6b3507cecfaa7c8f481abae0599f79409bf81b6b203328f765039c27135f6faf"} Oct 30 05:48:30.769906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.769877 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl"] Oct 30 05:48:30.774218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.774199 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.776349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.776324 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65wb9\" (UniqueName: \"kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.776479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.776363 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.776479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.776452 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.796011 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.795985 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl"] Oct 30 05:48:30.878135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.878100 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65wb9\" (UniqueName: \"kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.878135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.878138 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.878330 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.878190 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.878976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.878957 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.880466 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.880450 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:30.902579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:30.902545 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65wb9\" (UniqueName: \"kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9\") pod \"maas-api-revoke-sub-7181d5eb-9fdjl\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:31.087711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.087630 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:31.393280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.393252 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl"] Oct 30 05:48:31.394654 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:31.394625 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5dafc59_5f16_4d11_9bb4_266aef5ce9a3.slice/crio-3c8da8a532b268952bd061d210b0e3e410fc12bb3258b93c1fc31d292c43ab98 WatchSource:0}: Error finding container 3c8da8a532b268952bd061d210b0e3e410fc12bb3258b93c1fc31d292c43ab98: Status 404 returned error can't find the container with id 3c8da8a532b268952bd061d210b0e3e410fc12bb3258b93c1fc31d292c43ab98 Oct 30 05:48:31.479435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.479414 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:31.485589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.485569 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca\") pod \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " Oct 30 05:48:31.485751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.485737 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca\") pod \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " Oct 30 05:48:31.485834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.485822 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb\" (UniqueName: \"kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb\") pod \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\" (UID: \"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf\") " Oct 30 05:48:31.486081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.486055 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca" pod "bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" (UID: "bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:31.486243 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.486229 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:31.487571 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.487550 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca" pod "bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" (UID: "bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:31.487732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.487711 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb" pod "bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" (UID: "bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf"). InnerVolumeSpecName "kube-api-access-vdkzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:31.587231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.587195 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdkzb\" (UniqueName: \"kubernetes.io/projected/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-kube-api-access-vdkzb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:31.587231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.587232 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:31.745509 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.745472 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748"] Oct 30 05:48:31.746157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.746136 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" containerName="revoke-keys" Oct 30 05:48:31.746261 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.746157 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" containerName="revoke-keys" Oct 30 05:48:31.746345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.746329 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" containerName="revoke-keys" Oct 30 05:48:31.749977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.749955 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.780799 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.780768 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748"] Oct 30 05:48:31.789042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789012 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789134 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789243 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789279 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789307 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789407 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789498 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789550 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.789612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.789581 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfj9\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891415 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891506 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891650 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891685 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891715 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891743 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891808 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891848 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.892551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.891877 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfj9\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.893121 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.892678 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.893121 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.892949 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.898143 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.898118 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.900600 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.900569 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.901051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.901027 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.901194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.901160 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.901323 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.901291 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.916467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.916409 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfj9\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:31.923283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:31.923262 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token\") pod \"e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:32.043860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.043826 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf"] Oct 30 05:48:32.049423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.049396 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.067353 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.067328 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:32.077891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.077862 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf"] Oct 30 05:48:32.146044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.146010 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz"] Oct 30 05:48:32.152053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.152031 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.173610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.173077 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz"] Oct 30 05:48:32.205614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.205583 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.205834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.205804 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7kd\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.205967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.205840 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.205967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.205898 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.205967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.205929 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxbsj\" (UniqueName: \"kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.206140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.205977 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.206140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.206051 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.206140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.206095 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.206280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.206192 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.206280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.206229 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.206280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.206270 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.206419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.206309 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307111 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307191 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.307275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307211 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307238 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307265 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307363 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307397 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7kd\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307420 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.307633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307469 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307499 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxbsj\" (UniqueName: \"kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.307633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307567 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307646 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307683 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307728 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307919 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307889 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.307981 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.307964 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.308352 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.308327 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.308473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.308426 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.309809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.309786 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.310014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.309993 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.310131 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.310113 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.340142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.338179 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7kd\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.340142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.338286 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxbsj\" (UniqueName: \"kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj\") pod \"maas-api-revoke-sub-61fed433-t5tsz\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.343212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.343190 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token\") pod \"e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.347773 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.347744 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" containerID="d42ae00ec4024ff5dac2c3aa101cf2715b3bf287a854988d7c1af830019daffb" exitCode=0 Oct 30 05:48:32.347994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.347823 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" event={"ID":"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3","Type":"ContainerDied","Data":"d42ae00ec4024ff5dac2c3aa101cf2715b3bf287a854988d7c1af830019daffb"} Oct 30 05:48:32.347994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.347887 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" event={"ID":"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3","Type":"ContainerStarted","Data":"3c8da8a532b268952bd061d210b0e3e410fc12bb3258b93c1fc31d292c43ab98"} Oct 30 05:48:32.349628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.349607 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" Oct 30 05:48:32.349628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.349609 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr" event={"ID":"bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf","Type":"ContainerDied","Data":"027c5da16401b5a03c4ed6b7f4081824ad4b2865c23e0f182584dfa9c557dc96"} Oct 30 05:48:32.349774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.349655 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="027c5da16401b5a03c4ed6b7f4081824ad4b2865c23e0f182584dfa9c557dc96" Oct 30 05:48:32.352383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.352362 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml"] Oct 30 05:48:32.357757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.357741 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.363836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.363807 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:32.384440 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.384412 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml"] Oct 30 05:48:32.417101 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:32.416740 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod072249aa_e351_49d2_81d0_e20398bf9c15.slice/crio-948eb766c1c5a0562b4983ab2b1fe61bd2e65bde2c797b48b040bbad45af541e WatchSource:0}: Error finding container 948eb766c1c5a0562b4983ab2b1fe61bd2e65bde2c797b48b040bbad45af541e: Status 404 returned error can't find the container with id 948eb766c1c5a0562b4983ab2b1fe61bd2e65bde2c797b48b040bbad45af541e Oct 30 05:48:32.418678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.418224 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748"] Oct 30 05:48:32.421630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.419930 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:32.421630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.420000 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:32.421630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.420038 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:32.473681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.473652 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:32.515827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.515793 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.516260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.515938 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.516260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.515988 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.516260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.516031 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.516260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.516067 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.516260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.516184 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.524841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.516264 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f8c5h\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.524841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.516301 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.524841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.516356 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.617837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.617802 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.618004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.617911 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.618004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.617958 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.618004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.617998 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.618100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618031 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.618260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618139 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618583 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618838 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618952 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f8c5h\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618992 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.619046 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.619240 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.618977 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.619879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.619466 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.621599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.621576 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.621685 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.621633 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.644055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.643890 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.645573 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.645517 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f8c5h\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.675676 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.675602 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:32.697135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.697106 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf"] Oct 30 05:48:32.700829 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:32.700795 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f573711_e9e1_4c7d_8ed4_b4b6976062e4.slice/crio-061f1897118f37b1ed93a3a2435e99bad86d251ec9ef1c532b7b8bdf5b37730e WatchSource:0}: Error finding container 061f1897118f37b1ed93a3a2435e99bad86d251ec9ef1c532b7b8bdf5b37730e: Status 404 returned error can't find the container with id 061f1897118f37b1ed93a3a2435e99bad86d251ec9ef1c532b7b8bdf5b37730e Oct 30 05:48:32.703771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.703736 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:32.703865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.703831 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:32.703915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.703875 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:32.865547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:32.865466 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz"] Oct 30 05:48:32.871596 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:32.871568 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9ec65e1_f683_4166_824e_ccfa6bcce7c4.slice/crio-4851d1303630206a9f960989ca51865df9429046fb64e7141d580cbbf9ef6dd4 WatchSource:0}: Error finding container 4851d1303630206a9f960989ca51865df9429046fb64e7141d580cbbf9ef6dd4: Status 404 returned error can't find the container with id 4851d1303630206a9f960989ca51865df9429046fb64e7141d580cbbf9ef6dd4 Oct 30 05:48:33.001197 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.001144 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml"] Oct 30 05:48:33.001305 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:33.001239 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod271c4540_a7c9_4694_aec5_3f3326a35d92.slice/crio-e4e513d01f8ced2177aba6e9cf0088b01e36c39094163466edbda5fcfdceae92 WatchSource:0}: Error finding container e4e513d01f8ced2177aba6e9cf0088b01e36c39094163466edbda5fcfdceae92: Status 404 returned error can't find the container with id e4e513d01f8ced2177aba6e9cf0088b01e36c39094163466edbda5fcfdceae92 Oct 30 05:48:33.003774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.003744 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:33.003860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.003828 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:33.003913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.003895 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:33.412588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.411370 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" event={"ID":"072249aa-e351-49d2-81d0-e20398bf9c15","Type":"ContainerStarted","Data":"a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9"} Oct 30 05:48:33.412588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.411491 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" event={"ID":"072249aa-e351-49d2-81d0-e20398bf9c15","Type":"ContainerStarted","Data":"948eb766c1c5a0562b4983ab2b1fe61bd2e65bde2c797b48b040bbad45af541e"} Oct 30 05:48:33.414388 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.414356 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" event={"ID":"271c4540-a7c9-4694-aec5-3f3326a35d92","Type":"ContainerStarted","Data":"2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10"} Oct 30 05:48:33.414603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.414586 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" event={"ID":"271c4540-a7c9-4694-aec5-3f3326a35d92","Type":"ContainerStarted","Data":"e4e513d01f8ced2177aba6e9cf0088b01e36c39094163466edbda5fcfdceae92"} Oct 30 05:48:33.418286 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.418255 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" event={"ID":"c9ec65e1-f683-4166-824e-ccfa6bcce7c4","Type":"ContainerStarted","Data":"4851d1303630206a9f960989ca51865df9429046fb64e7141d580cbbf9ef6dd4"} Oct 30 05:48:33.421438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.421413 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" event={"ID":"6f573711-e9e1-4c7d-8ed4-b4b6976062e4","Type":"ContainerStarted","Data":"89a0b1520331f9e34a35cb740fb9f3905f312fb5c7ed063fbd142115bf734fbe"} Oct 30 05:48:33.421631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.421612 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" event={"ID":"6f573711-e9e1-4c7d-8ed4-b4b6976062e4","Type":"ContainerStarted","Data":"061f1897118f37b1ed93a3a2435e99bad86d251ec9ef1c532b7b8bdf5b37730e"} Oct 30 05:48:33.486552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.468501 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podStartSLOduration=2.468482877 podStartE2EDuration="2.468482877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:33.468023069 +0000 UTC m=+1020.043710126" watchObservedRunningTime="2026-10-30 05:48:33.468482877 +0000 UTC m=+1020.044169932" Oct 30 05:48:33.614559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.583471 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podStartSLOduration=1.583450706 podStartE2EDuration="1.583450706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:33.556986922 +0000 UTC m=+1020.132673977" watchObservedRunningTime="2026-10-30 05:48:33.583450706 +0000 UTC m=+1020.159137765" Oct 30 05:48:33.640037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.634619 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podStartSLOduration=1.634599536 podStartE2EDuration="1.634599536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:33.626026424 +0000 UTC m=+1020.201713480" watchObservedRunningTime="2026-10-30 05:48:33.634599536 +0000 UTC m=+1020.210286590" Oct 30 05:48:33.691714 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:33.691626 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:34.010190 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.010157 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:34.049745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.049717 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9\" (UniqueName: \"kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9\") pod \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " Oct 30 05:48:34.050010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.049994 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca\") pod \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " Oct 30 05:48:34.050189 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.050177 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca\") pod \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\" (UID: \"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3\") " Oct 30 05:48:34.050672 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.050446 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca" pod "f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" (UID: "f5dafc59-5f16-4d11-9bb4-266aef5ce9a3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:34.051933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.051905 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca" pod "f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" (UID: "f5dafc59-5f16-4d11-9bb4-266aef5ce9a3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:34.052024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.051993 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9" pod "f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" (UID: "f5dafc59-5f16-4d11-9bb4-266aef5ce9a3"). InnerVolumeSpecName "kube-api-access-65wb9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:34.052160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.052143 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:34.052216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.052163 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:34.052216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.052172 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65wb9\" (UniqueName: \"kubernetes.io/projected/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3-kube-api-access-65wb9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:34.133770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.128624 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:34.133770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.128749 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.133770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.128808 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:48:34.385423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.367838 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:34.402619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.401363 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.402619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.401423 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:48:34.440760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.440734 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" Oct 30 05:48:34.441473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.440889 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl" event={"ID":"f5dafc59-5f16-4d11-9bb4-266aef5ce9a3","Type":"ContainerDied","Data":"3c8da8a532b268952bd061d210b0e3e410fc12bb3258b93c1fc31d292c43ab98"} Oct 30 05:48:34.441626 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.441500 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3c8da8a532b268952bd061d210b0e3e410fc12bb3258b93c1fc31d292c43ab98" Oct 30 05:48:34.442579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.442554 5565 generic.go:356] "Generic (PLEG): container finished" podID="c9ec65e1-f683-4166-824e-ccfa6bcce7c4" containerID="76521c544530547bf25237536cd1c8e33178ace3a08cefeadf834fabcaa05e8f" exitCode=0 Oct 30 05:48:34.443640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.443483 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" event={"ID":"c9ec65e1-f683-4166-824e-ccfa6bcce7c4","Type":"ContainerDied","Data":"76521c544530547bf25237536cd1c8e33178ace3a08cefeadf834fabcaa05e8f"} Oct 30 05:48:34.692605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.691690 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:48:34.692605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:34.691753 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:48:35.072552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.068870 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.072552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.068929 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:48:35.369611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.366871 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.369611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.366928 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:48:35.712550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.680901 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:48:35.712550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.680972 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:48:36.014607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:35.965685 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:36.050234 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.050190 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca\") pod \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " Oct 30 05:48:36.050406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.050240 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca\") pod \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " Oct 30 05:48:36.050465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.050443 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj\" (UniqueName: \"kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj\") pod \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\" (UID: \"c9ec65e1-f683-4166-824e-ccfa6bcce7c4\") " Oct 30 05:48:36.050958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.050925 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca" pod "c9ec65e1-f683-4166-824e-ccfa6bcce7c4" (UID: "c9ec65e1-f683-4166-824e-ccfa6bcce7c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:36.051341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.051309 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:36.118547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.116523 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:36.118547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.116609 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:48:36.118547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.117248 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj" pod "c9ec65e1-f683-4166-824e-ccfa6bcce7c4" (UID: "c9ec65e1-f683-4166-824e-ccfa6bcce7c4"). InnerVolumeSpecName "kube-api-access-mxbsj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:36.138577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.118983 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca" pod "c9ec65e1-f683-4166-824e-ccfa6bcce7c4" (UID: "c9ec65e1-f683-4166-824e-ccfa6bcce7c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:36.152491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.152458 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxbsj\" (UniqueName: \"kubernetes.io/projected/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-kube-api-access-mxbsj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:36.152491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.152486 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c9ec65e1-f683-4166-824e-ccfa6bcce7c4-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:36.385370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.379941 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:48:36.385370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.380001 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:48:36.528630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.518222 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" event={"ID":"c9ec65e1-f683-4166-824e-ccfa6bcce7c4","Type":"ContainerDied","Data":"4851d1303630206a9f960989ca51865df9429046fb64e7141d580cbbf9ef6dd4"} Oct 30 05:48:36.528630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.518321 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4851d1303630206a9f960989ca51865df9429046fb64e7141d580cbbf9ef6dd4" Oct 30 05:48:36.528630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.518394 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz" Oct 30 05:48:36.727914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.727821 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:48:36.728739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:36.728705 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:48:37.084557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:37.069767 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.145103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:37.145057 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:48:37.365031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:37.364872 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.365031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:37.364931 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:48:37.683485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:37.682988 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:48:37.683485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:37.683045 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:48:38.077935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:38.077894 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.081633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:38.081592 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:48:38.441594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:38.441499 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.441758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:38.441694 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:48:38.681252 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:38.681204 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:48:38.681570 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:38.681515 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:48:39.084550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.081632 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.084550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.081694 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 05:48:39.100557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.100515 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng"] Oct 30 05:48:39.101641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.101621 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" containerName="revoke-keys" Oct 30 05:48:39.101780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.101768 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" containerName="revoke-keys" Oct 30 05:48:39.101922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.101912 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c9ec65e1-f683-4166-824e-ccfa6bcce7c4" containerName="revoke-keys" Oct 30 05:48:39.101991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.101982 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9ec65e1-f683-4166-824e-ccfa6bcce7c4" containerName="revoke-keys" Oct 30 05:48:39.102212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.102200 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9ec65e1-f683-4166-824e-ccfa6bcce7c4" containerName="revoke-keys" Oct 30 05:48:39.102306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.102297 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" containerName="revoke-keys" Oct 30 05:48:39.106802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.106783 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs"] Oct 30 05:48:39.106935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.106909 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.111641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.111620 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.124634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.124611 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:39.132940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.132918 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:39.133162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.133121 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:39.138560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.138501 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:48:39.138935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.138916 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng"] Oct 30 05:48:39.145189 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.145164 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs"] Oct 30 05:48:39.151111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.151084 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zrcb\" (UniqueName: \"kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.151352 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.151332 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.151517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.151500 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.151780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.151760 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.194827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.152403 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.194998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.194967 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fktb\" (UniqueName: \"kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.296517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.296479 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.296712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.296561 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.296712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.296626 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.296847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.296785 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.296901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.296850 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fktb\" (UniqueName: \"kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.296901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.296883 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zrcb\" (UniqueName: \"kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.297329 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.297297 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.297905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.297878 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.299364 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.299335 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.299473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.299452 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.329669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.329641 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zrcb\" (UniqueName: \"kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb\") pod \"maas-api-revoke-sub-7b996d00-8zvzs\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.355782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.355726 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fktb\" (UniqueName: \"kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb\") pod \"maas-api-revoke-sub-d64f561f-bp8ng\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.364770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.364745 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.364862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.364795 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 05:48:39.440066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.440031 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:39.450594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.450570 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:39.676802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.676715 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 05:48:39.676802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.676780 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 05:48:39.868789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.868716 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs"] Oct 30 05:48:39.869952 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:39.869927 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421b838d_7dbd_4ca9_9046_25bfd4abe129.slice/crio-af3e6a39e0902db331f435b431ea4d94377e2abe3b5e2993895a1c9accf10480 WatchSource:0}: Error finding container af3e6a39e0902db331f435b431ea4d94377e2abe3b5e2993895a1c9accf10480: Status 404 returned error can't find the container with id af3e6a39e0902db331f435b431ea4d94377e2abe3b5e2993895a1c9accf10480 Oct 30 05:48:39.898986 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:39.898954 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5703bb0b_42cb_4e32_8045_9103f17cf1ce.slice/crio-83d04641c7cac837b0e3b5d7bd62ba4a94d9896fd238705fbc0ab281117b5f18 WatchSource:0}: Error finding container 83d04641c7cac837b0e3b5d7bd62ba4a94d9896fd238705fbc0ab281117b5f18: Status 404 returned error can't find the container with id 83d04641c7cac837b0e3b5d7bd62ba4a94d9896fd238705fbc0ab281117b5f18 Oct 30 05:48:39.901814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:39.901787 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng"] Oct 30 05:48:40.670608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.664648 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:40.670608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.666029 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:40.673774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.673710 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 05:48:40.679850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.675450 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" event={"ID":"421b838d-7dbd-4ca9-9046-25bfd4abe129","Type":"ContainerStarted","Data":"cb8dee854f0998eafeae7b8e6c3881a15dfed63e3924dd3f22496f051394b370"} Oct 30 05:48:40.679850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.675499 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" event={"ID":"421b838d-7dbd-4ca9-9046-25bfd4abe129","Type":"ContainerStarted","Data":"af3e6a39e0902db331f435b431ea4d94377e2abe3b5e2993895a1c9accf10480"} Oct 30 05:48:40.679850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.679145 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" event={"ID":"5703bb0b-42cb-4e32-8045-9103f17cf1ce","Type":"ContainerStarted","Data":"4fabdab30c71712b0dc9713728380e2edb3630865fd11b87bb187da5cd974bbd"} Oct 30 05:48:40.679850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.679181 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" event={"ID":"5703bb0b-42cb-4e32-8045-9103f17cf1ce","Type":"ContainerStarted","Data":"83d04641c7cac837b0e3b5d7bd62ba4a94d9896fd238705fbc0ab281117b5f18"} Oct 30 05:48:40.761268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.761205 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" podStartSLOduration=1.761186014 podStartE2EDuration="1.761186014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:40.757314395 +0000 UTC m=+1027.333001451" watchObservedRunningTime="2026-10-30 05:48:40.761186014 +0000 UTC m=+1027.336873068" Oct 30 05:48:40.851331 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.851268 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" podStartSLOduration=1.851250549 podStartE2EDuration="1.851250549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:40.844972856 +0000 UTC m=+1027.420659911" watchObservedRunningTime="2026-10-30 05:48:40.851250549 +0000 UTC m=+1027.426937607" Oct 30 05:48:40.922069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.921218 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:40.922069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.921834 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:40.924463 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:40.922761 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 05:48:41.043385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.043356 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:41.044139 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.044112 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:41.045313 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.045285 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:48:41.687740 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.687705 5565 generic.go:356] "Generic (PLEG): container finished" podID="421b838d-7dbd-4ca9-9046-25bfd4abe129" containerID="cb8dee854f0998eafeae7b8e6c3881a15dfed63e3924dd3f22496f051394b370" exitCode=0 Oct 30 05:48:41.688140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.687795 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" event={"ID":"421b838d-7dbd-4ca9-9046-25bfd4abe129","Type":"ContainerDied","Data":"cb8dee854f0998eafeae7b8e6c3881a15dfed63e3924dd3f22496f051394b370"} Oct 30 05:48:41.689277 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.689253 5565 generic.go:356] "Generic (PLEG): container finished" podID="5703bb0b-42cb-4e32-8045-9103f17cf1ce" containerID="4fabdab30c71712b0dc9713728380e2edb3630865fd11b87bb187da5cd974bbd" exitCode=0 Oct 30 05:48:41.689465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:41.689331 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" event={"ID":"5703bb0b-42cb-4e32-8045-9103f17cf1ce","Type":"ContainerDied","Data":"4fabdab30c71712b0dc9713728380e2edb3630865fd11b87bb187da5cd974bbd"} Oct 30 05:48:42.861463 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.861429 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:42.904748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.902192 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:42.966202 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.966114 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca\") pod \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " Oct 30 05:48:42.966202 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.966172 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb\" (UniqueName: \"kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb\") pod \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " Oct 30 05:48:42.966542 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.966495 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca" pod "5703bb0b-42cb-4e32-8045-9103f17cf1ce" (UID: "5703bb0b-42cb-4e32-8045-9103f17cf1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:42.966642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.966510 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca\") pod \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\" (UID: \"5703bb0b-42cb-4e32-8045-9103f17cf1ce\") " Oct 30 05:48:42.967208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.967180 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:42.968503 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.968465 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb" pod "5703bb0b-42cb-4e32-8045-9103f17cf1ce" (UID: "5703bb0b-42cb-4e32-8045-9103f17cf1ce"). InnerVolumeSpecName "kube-api-access-9fktb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:42.968897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:42.968873 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca" pod "5703bb0b-42cb-4e32-8045-9103f17cf1ce" (UID: "5703bb0b-42cb-4e32-8045-9103f17cf1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:43.068324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068294 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca\") pod \"421b838d-7dbd-4ca9-9046-25bfd4abe129\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " Oct 30 05:48:43.068473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068352 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca\") pod \"421b838d-7dbd-4ca9-9046-25bfd4abe129\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " Oct 30 05:48:43.068473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068398 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb\" (UniqueName: \"kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb\") pod \"421b838d-7dbd-4ca9-9046-25bfd4abe129\" (UID: \"421b838d-7dbd-4ca9-9046-25bfd4abe129\") " Oct 30 05:48:43.068748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068722 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca" pod "421b838d-7dbd-4ca9-9046-25bfd4abe129" (UID: "421b838d-7dbd-4ca9-9046-25bfd4abe129"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:43.068964 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068946 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5703bb0b-42cb-4e32-8045-9103f17cf1ce-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:43.069047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068964 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fktb\" (UniqueName: \"kubernetes.io/projected/5703bb0b-42cb-4e32-8045-9103f17cf1ce-kube-api-access-9fktb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:43.069047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.068974 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:43.070506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.070480 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb" pod "421b838d-7dbd-4ca9-9046-25bfd4abe129" (UID: "421b838d-7dbd-4ca9-9046-25bfd4abe129"). InnerVolumeSpecName "kube-api-access-5zrcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:43.070506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.070497 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca" pod "421b838d-7dbd-4ca9-9046-25bfd4abe129" (UID: "421b838d-7dbd-4ca9-9046-25bfd4abe129"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:43.170138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.170105 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/421b838d-7dbd-4ca9-9046-25bfd4abe129-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:43.170138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.170140 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zrcb\" (UniqueName: \"kubernetes.io/projected/421b838d-7dbd-4ca9-9046-25bfd4abe129-kube-api-access-5zrcb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:43.704864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.704833 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:48:43.705027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.704831 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" event={"ID":"421b838d-7dbd-4ca9-9046-25bfd4abe129","Type":"ContainerDied","Data":"af3e6a39e0902db331f435b431ea4d94377e2abe3b5e2993895a1c9accf10480"} Oct 30 05:48:43.705027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.704967 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af3e6a39e0902db331f435b431ea4d94377e2abe3b5e2993895a1c9accf10480" Oct 30 05:48:43.706619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.706591 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" Oct 30 05:48:43.706619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.706594 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng" event={"ID":"5703bb0b-42cb-4e32-8045-9103f17cf1ce","Type":"ContainerDied","Data":"83d04641c7cac837b0e3b5d7bd62ba4a94d9896fd238705fbc0ab281117b5f18"} Oct 30 05:48:43.706823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:43.706633 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83d04641c7cac837b0e3b5d7bd62ba4a94d9896fd238705fbc0ab281117b5f18" Oct 30 05:48:44.891670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:44.891639 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:48:47.463551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.463403 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:48:47.500551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.492198 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:47.500551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.493903 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:48:47.500551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.494928 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.513240 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd"] Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.514130 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5703bb0b-42cb-4e32-8045-9103f17cf1ce" containerName="revoke-keys" Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.514148 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5703bb0b-42cb-4e32-8045-9103f17cf1ce" containerName="revoke-keys" Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.514170 5565 cpu_manager.go:414] "containerMap: removing container" podUID="421b838d-7dbd-4ca9-9046-25bfd4abe129" containerName="revoke-keys" Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.514177 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="421b838d-7dbd-4ca9-9046-25bfd4abe129" containerName="revoke-keys" Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.514370 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5703bb0b-42cb-4e32-8045-9103f17cf1ce" containerName="revoke-keys" Oct 30 05:48:47.515551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.514382 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="421b838d-7dbd-4ca9-9046-25bfd4abe129" containerName="revoke-keys" Oct 30 05:48:47.559565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.539153 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": context deadline exceeded" Oct 30 05:48:47.599877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.599303 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:48:47.667570 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.637467 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.768937 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769046 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769077 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769136 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769206 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769244 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769285 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.776410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.769478 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.796615 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.794286 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:47.840299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.839450 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn6k8\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.902554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.887761 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd"] Oct 30 05:48:47.934557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.912616 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:47.946552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.941472 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.958843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.958813 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.959499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.959478 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.960029 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.960011 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.960491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.960470 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.961355 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.961334 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.963198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.963176 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.963657 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.963639 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn6k8\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.964119 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.964098 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.972147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.959926 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.972316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.961742 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.972434 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.947135 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.972579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.963560 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.972763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.958423 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 05:48:47.972901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.961266 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:47.997385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:47.997346 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.021579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.009336 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:48.022430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.022399 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:48.023253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.023229 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn6k8\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:48.045591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.045509 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy\") pod \"e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:48.102761 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:48:48.102721 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:48:48.214830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.214793 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:48.486980 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.486948 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.489510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.489487 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:48:48.494668 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.491002 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.502657 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.494884 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.522090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.520650 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:48:48.522090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.494911 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:48:48.541992 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.495012 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.543292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.494943 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.544188 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.544166 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:48:48.544418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.544402 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:48:48.614565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.608325 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:48:48.619706 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.619673 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:48:48.658844 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.486855 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.849660 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.849607 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 05:48:48.857182 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.857152 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:48:48.936719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.936666 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:48.944604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.944578 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:48:48.999404 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.999364 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:48:48.999977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:48.999948 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": context deadline exceeded" Oct 30 05:48:49.000223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.000202 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:48:49.517551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.512382 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:49.517551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.512808 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:49.517551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.513418 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:48:49.559560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.528024 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:48:49.621553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.619925 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:48:49.621553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.620546 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:48:49.675551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.675177 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 05:48:49.879609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.872380 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:49.879609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.872639 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:48:49.952555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.948097 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:49.975563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:49.975052 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:48:50.293555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.258334 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:50.295034 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.294995 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:48:50.562399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.562300 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:48:50.563334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.563300 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:50.563781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.563746 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:48:50.620666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.620615 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:50.676092 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:48:50.669283 5565 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.027s" Oct 30 05:48:50.676092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.669355 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:48:50.676092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.669417 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:48:50.676092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.669460 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:48:50.676092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.669495 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:48:50.676092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.675598 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:48:50.710568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.709436 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:48:50.772578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.772549 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:48:50.781707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.780430 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:48:50.785584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.785562 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:48:50.792934 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.792911 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:48:50.796322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.796199 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:48:50.798648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.797109 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:48:50.798648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.798193 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:48:50.915831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.909261 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd"] Oct 30 05:48:50.915831 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:50.911054 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37723510_9540_4794_b253_9e85fc0f6846.slice/crio-ba1e192ec5a1b2b169dd060e564101217a6da650d6673faa5a978e3437a7e96b WatchSource:0}: Error finding container ba1e192ec5a1b2b169dd060e564101217a6da650d6673faa5a978e3437a7e96b: Status 404 returned error can't find the container with id ba1e192ec5a1b2b169dd060e564101217a6da650d6673faa5a978e3437a7e96b Oct 30 05:48:50.915831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.914752 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:50.915831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.914833 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:50.915831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:50.914877 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:48:51.317758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:51.317722 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" event={"ID":"37723510-9540-4794-b253-9e85fc0f6846","Type":"ContainerStarted","Data":"2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e"} Oct 30 05:48:51.317916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:51.317783 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" event={"ID":"37723510-9540-4794-b253-9e85fc0f6846","Type":"ContainerStarted","Data":"ba1e192ec5a1b2b169dd060e564101217a6da650d6673faa5a978e3437a7e96b"} Oct 30 05:48:51.386441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:51.386379 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podStartSLOduration=4.386365875 podStartE2EDuration="4.386365875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:51.38559737 +0000 UTC m=+1037.961284424" watchObservedRunningTime="2026-10-30 05:48:51.386365875 +0000 UTC m=+1037.962052928" Oct 30 05:48:52.216305 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:52.216269 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:52.217755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:52.217732 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:52.217858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:52.217779 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:53.216857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.216771 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:53.216857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.216835 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:53.646675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.646644 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4"] Oct 30 05:48:53.650603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.650580 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.678821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.678795 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:48:53.691852 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.691824 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:48:53.691852 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.691839 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:48:53.692002 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.691924 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:48:53.695966 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.695942 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4"] Oct 30 05:48:53.719158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.719133 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws"] Oct 30 05:48:53.723305 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.723287 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.748469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.748434 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws"] Oct 30 05:48:53.825849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.825763 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.825849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.825831 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j999j\" (UniqueName: \"kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.826038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.825858 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.826038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.825899 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.826038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.826028 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwrl4\" (UniqueName: \"kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.826138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.826048 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.927688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.927605 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.927688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.927660 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j999j\" (UniqueName: \"kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.927924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.927785 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.927924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.927867 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.928095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.928077 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwrl4\" (UniqueName: \"kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.928133 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.928119 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.928414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.928395 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.928737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.928713 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.930562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.930539 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.930562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.930519 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.952109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.952064 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwrl4\" (UniqueName: \"kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4\") pod \"maas-api-revoke-sub-0cf7d6b8-zr6ws\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:53.980212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.952606 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j999j\" (UniqueName: \"kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j\") pod \"maas-api-revoke-sub-c9ea81c2-lpng4\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:53.992740 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:53.992713 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:54.037828 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:54.037789 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:54.220713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:54.220668 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:54.220997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:54.220746 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:54.345085 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:54.345052 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4"] Oct 30 05:48:54.447759 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:54.447723 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93d6d5c3_8cac_4e2d_aab2_94857cae53c1.slice/crio-ed795c86dd42da299d96964927e18ee811a5e937dd8fd2c7af09e3c0abe22625 WatchSource:0}: Error finding container ed795c86dd42da299d96964927e18ee811a5e937dd8fd2c7af09e3c0abe22625: Status 404 returned error can't find the container with id ed795c86dd42da299d96964927e18ee811a5e937dd8fd2c7af09e3c0abe22625 Oct 30 05:48:54.448311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:54.448229 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws"] Oct 30 05:48:55.216213 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.216124 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:55.216213 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.216178 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:55.349224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.349189 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" event={"ID":"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8","Type":"ContainerStarted","Data":"ae87a1d2fa5473f980bbe9e610f61e7387eabaddb9222f95cb09aff74fdad68c"} Oct 30 05:48:55.349649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.349243 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" event={"ID":"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8","Type":"ContainerStarted","Data":"0b41b3fcac83862ed168e649533be6712f6684b5304870e82ed2468f5fd08057"} Oct 30 05:48:55.350962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.350937 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" event={"ID":"93d6d5c3-8cac-4e2d-aab2-94857cae53c1","Type":"ContainerStarted","Data":"204ff350ed5eff73363aa5b017801f0212e86e81328f6869e509d68c8438c723"} Oct 30 05:48:55.351080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.350976 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" event={"ID":"93d6d5c3-8cac-4e2d-aab2-94857cae53c1","Type":"ContainerStarted","Data":"ed795c86dd42da299d96964927e18ee811a5e937dd8fd2c7af09e3c0abe22625"} Oct 30 05:48:55.395741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.395693 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" podStartSLOduration=2.3956792399999998 podStartE2EDuration="2.39567924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:55.392094802 +0000 UTC m=+1041.967781857" watchObservedRunningTime="2026-10-30 05:48:55.39567924 +0000 UTC m=+1041.971366294" Oct 30 05:48:55.435572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:55.435509 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" podStartSLOduration=2.435495884 podStartE2EDuration="2.435495884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:55.432724239 +0000 UTC m=+1042.008411296" watchObservedRunningTime="2026-10-30 05:48:55.435495884 +0000 UTC m=+1042.011182953" Oct 30 05:48:56.215999 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.215968 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:56.216212 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.216023 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:56.383061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.383022 5565 generic.go:356] "Generic (PLEG): container finished" podID="6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" containerID="ae87a1d2fa5473f980bbe9e610f61e7387eabaddb9222f95cb09aff74fdad68c" exitCode=0 Oct 30 05:48:56.383511 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.383101 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" event={"ID":"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8","Type":"ContainerDied","Data":"ae87a1d2fa5473f980bbe9e610f61e7387eabaddb9222f95cb09aff74fdad68c"} Oct 30 05:48:56.385217 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.385186 5565 generic.go:356] "Generic (PLEG): container finished" podID="93d6d5c3-8cac-4e2d-aab2-94857cae53c1" containerID="204ff350ed5eff73363aa5b017801f0212e86e81328f6869e509d68c8438c723" exitCode=0 Oct 30 05:48:56.385341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.385225 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" event={"ID":"93d6d5c3-8cac-4e2d-aab2-94857cae53c1","Type":"ContainerDied","Data":"204ff350ed5eff73363aa5b017801f0212e86e81328f6869e509d68c8438c723"} Oct 30 05:48:56.481974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.481878 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s"] Oct 30 05:48:56.487612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.487505 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.509396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.509363 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"default-dockercfg-nchgr\"" Oct 30 05:48:56.510423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.510398 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"openshift-service-ca.crt\"" Oct 30 05:48:56.511363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.511316 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"kube-root-ca.crt\"" Oct 30 05:48:56.511488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.511470 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-be78f2\"/\"distinct-w5-be78f2-kserve-self-signed-certs\"" Oct 30 05:48:56.519869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.519841 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s"] Oct 30 05:48:56.585373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.585336 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.585516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.585396 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.585591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.585538 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mj4jt\" (UniqueName: \"kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.585591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.585580 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.585708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.585685 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.585822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.585809 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.687083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.686959 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.687083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.687078 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.687366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.687123 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.687366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.687177 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mj4jt\" (UniqueName: \"kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.687366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.687224 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.687366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.687306 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.688021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.687992 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.688297 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.688274 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.688436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.688413 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.690598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.690582 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.691051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.690979 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.711764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.711741 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mj4jt\" (UniqueName: \"kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt\") pod \"distinct-w5-be78f2-kserve-7cb54887c8-5g64s\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:56.810536 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:56.810496 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:57.163182 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.163152 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s"] Oct 30 05:48:57.163643 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:48:57.163616 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b3adc8_3939_4a61_b5c1_8c69f82c3361.slice/crio-f615b631b9f4037da07876e0ffc717353f9a0afc5ce73f80b1db802c401ffcff WatchSource:0}: Error finding container f615b631b9f4037da07876e0ffc717353f9a0afc5ce73f80b1db802c401ffcff: Status 404 returned error can't find the container with id f615b631b9f4037da07876e0ffc717353f9a0afc5ce73f80b1db802c401ffcff Oct 30 05:48:57.216320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.216295 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:57.216454 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.216347 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:57.395883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.395262 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" event={"ID":"d4b3adc8-3939-4a61-b5c1-8c69f82c3361","Type":"ContainerStarted","Data":"15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf"} Oct 30 05:48:57.395883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.395318 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" event={"ID":"d4b3adc8-3939-4a61-b5c1-8c69f82c3361","Type":"ContainerStarted","Data":"f615b631b9f4037da07876e0ffc717353f9a0afc5ce73f80b1db802c401ffcff"} Oct 30 05:48:57.440755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.440654 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podStartSLOduration=1.4406402919999999 podStartE2EDuration="1.440640292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:48:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:48:57.437799057 +0000 UTC m=+1044.013486111" watchObservedRunningTime="2026-10-30 05:48:57.440640292 +0000 UTC m=+1044.016327375" Oct 30 05:48:57.545542 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.545502 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:57.611944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.611912 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.710000 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4\" (UniqueName: \"kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4\") pod \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.710111 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca\") pod \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.710164 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca\") pod \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.710282 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca\") pod \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.710326 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca\") pod \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\" (UID: \"93d6d5c3-8cac-4e2d-aab2-94857cae53c1\") " Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.710413 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j\" (UniqueName: \"kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j\") pod \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\" (UID: \"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8\") " Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.712441 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca" pod "93d6d5c3-8cac-4e2d-aab2-94857cae53c1" (UID: "93d6d5c3-8cac-4e2d-aab2-94857cae53c1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.713370 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4" pod "93d6d5c3-8cac-4e2d-aab2-94857cae53c1" (UID: "93d6d5c3-8cac-4e2d-aab2-94857cae53c1"). InnerVolumeSpecName "kube-api-access-kwrl4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.714350 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca" pod "6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" (UID: "6ec3ab3e-29c3-4771-a4f8-ad5467873cd8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.714463 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca" pod "93d6d5c3-8cac-4e2d-aab2-94857cae53c1" (UID: "93d6d5c3-8cac-4e2d-aab2-94857cae53c1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.714478 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j" pod "6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" (UID: "6ec3ab3e-29c3-4771-a4f8-ad5467873cd8"). InnerVolumeSpecName "kube-api-access-j999j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:48:57.716877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.715770 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca" pod "6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" (UID: "6ec3ab3e-29c3-4771-a4f8-ad5467873cd8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:48:57.811482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.811448 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:48:57.812294 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.812266 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:57.812294 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.812294 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:57.812499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.812305 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j999j\" (UniqueName: \"kubernetes.io/projected/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-kube-api-access-j999j\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:57.812499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.812315 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwrl4\" (UniqueName: \"kubernetes.io/projected/93d6d5c3-8cac-4e2d-aab2-94857cae53c1-kube-api-access-kwrl4\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:57.812499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.812327 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:57.812499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:57.812335 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:48:58.215801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.215773 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body= Oct 30 05:48:58.215995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.215825 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" Oct 30 05:48:58.405680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.405643 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" Oct 30 05:48:58.405680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.405656 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4" event={"ID":"6ec3ab3e-29c3-4771-a4f8-ad5467873cd8","Type":"ContainerDied","Data":"0b41b3fcac83862ed168e649533be6712f6684b5304870e82ed2468f5fd08057"} Oct 30 05:48:58.406163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.405733 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b41b3fcac83862ed168e649533be6712f6684b5304870e82ed2468f5fd08057" Oct 30 05:48:58.407177 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.407156 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" Oct 30 05:48:58.407310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.407186 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws" event={"ID":"93d6d5c3-8cac-4e2d-aab2-94857cae53c1","Type":"ContainerDied","Data":"ed795c86dd42da299d96964927e18ee811a5e937dd8fd2c7af09e3c0abe22625"} Oct 30 05:48:58.407310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:58.407230 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed795c86dd42da299d96964927e18ee811a5e937dd8fd2c7af09e3c0abe22625" Oct 30 05:48:59.220442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:59.220412 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:59.220844 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:59.220823 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:48:59.221558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:48:59.221508 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:49:02.628389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.628356 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2"] Oct 30 05:49:02.629090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.629073 5565 cpu_manager.go:414] "containerMap: removing container" podUID="93d6d5c3-8cac-4e2d-aab2-94857cae53c1" containerName="revoke-keys" Oct 30 05:49:02.629142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.629092 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93d6d5c3-8cac-4e2d-aab2-94857cae53c1" containerName="revoke-keys" Oct 30 05:49:02.629142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.629134 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" containerName="revoke-keys" Oct 30 05:49:02.629142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.629141 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" containerName="revoke-keys" Oct 30 05:49:02.629252 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.629242 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="93d6d5c3-8cac-4e2d-aab2-94857cae53c1" containerName="revoke-keys" Oct 30 05:49:02.629290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.629261 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" containerName="revoke-keys" Oct 30 05:49:02.634278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.634260 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.663038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.663016 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2"] Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.673772 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.673819 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.673912 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.673969 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98zgw\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.673998 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.674065 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.674161 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.674204 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.681347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.674252 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775649 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775681 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775732 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775767 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98zgw\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775785 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775832 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.775912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775900 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775939 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.775982 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.776101 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.776442 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.776487 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.776502 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.776577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.776514 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.778375 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.778349 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.778579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.778559 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.799905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.799877 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.800132 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.800112 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98zgw\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:02.947944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:02.947885 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:03.280710 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:03.280681 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56d49fe1_2070_4e60_90b3_4c077784c7ce.slice/crio-fb150b7b12a1de878d076714db84478e146805b1ac78e3e9d7a91bf27073903c WatchSource:0}: Error finding container fb150b7b12a1de878d076714db84478e146805b1ac78e3e9d7a91bf27073903c: Status 404 returned error can't find the container with id fb150b7b12a1de878d076714db84478e146805b1ac78e3e9d7a91bf27073903c Oct 30 05:49:03.281793 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.281768 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2"] Oct 30 05:49:03.283270 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.283242 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:03.283345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.283304 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:03.283345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.283333 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:03.452055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.452021 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" event={"ID":"56d49fe1-2070-4e60-90b3-4c077784c7ce","Type":"ContainerStarted","Data":"e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1"} Oct 30 05:49:03.452259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.452083 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" event={"ID":"56d49fe1-2070-4e60-90b3-4c077784c7ce","Type":"ContainerStarted","Data":"fb150b7b12a1de878d076714db84478e146805b1ac78e3e9d7a91bf27073903c"} Oct 30 05:49:03.504043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.503995 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podStartSLOduration=1.5039817979999999 podStartE2EDuration="1.503981798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:03.499384367 +0000 UTC m=+1050.075071433" watchObservedRunningTime="2026-10-30 05:49:03.503981798 +0000 UTC m=+1050.079668851" Oct 30 05:49:03.947994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:03.947961 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:04.950118 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:04.950082 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:49:04.950507 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:04.950156 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:49:05.948538 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:05.948492 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:49:05.948712 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:05.948577 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:49:06.749624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.749592 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc"] Oct 30 05:49:06.774837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.774810 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.789549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.784477 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc"] Oct 30 05:49:06.792953 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.792931 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:06.804029 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.804005 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:06.804127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.804073 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:06.804127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.804121 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:06.811591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.811567 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:49:06.825762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.825740 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:49:06.834547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.834505 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.834659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.834640 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.834792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.834774 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r9vk\" (UniqueName: \"kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.936024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.935996 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.936193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.936072 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r9vk\" (UniqueName: \"kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.936255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.936238 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.936899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.936877 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.938588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.938565 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:06.948647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.948625 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:49:06.948761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.948667 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:49:06.980754 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:06.980727 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r9vk\" (UniqueName: \"kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk\") pod \"maas-api-revoke-sub-bc6b5c6a-wzdkc\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:07.097178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:07.097148 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:07.411853 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:07.411810 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e3e226_ff88_4e30_a0d5_2e1d3968092a.slice/crio-9cdfb6fecc834beba865d5fa766762e7499f8b7a69d24a942e580b7a2972603c WatchSource:0}: Error finding container 9cdfb6fecc834beba865d5fa766762e7499f8b7a69d24a942e580b7a2972603c: Status 404 returned error can't find the container with id 9cdfb6fecc834beba865d5fa766762e7499f8b7a69d24a942e580b7a2972603c Oct 30 05:49:07.412973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:07.412953 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc"] Oct 30 05:49:07.519928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:07.519893 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" event={"ID":"93e3e226-ff88-4e30-a0d5-2e1d3968092a","Type":"ContainerStarted","Data":"9cdfb6fecc834beba865d5fa766762e7499f8b7a69d24a942e580b7a2972603c"} Oct 30 05:49:07.532398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:07.532361 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:49:07.948317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:07.948244 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:49:07.948317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:07.948293 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:49:08.532047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:08.532014 5565 generic.go:356] "Generic (PLEG): container finished" podID="93e3e226-ff88-4e30-a0d5-2e1d3968092a" containerID="d72e278259d8c2c34bb186146085122a66e79ed90ea1e9a95f912ce3c1a41e07" exitCode=0 Oct 30 05:49:08.532194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:08.532095 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" event={"ID":"93e3e226-ff88-4e30-a0d5-2e1d3968092a","Type":"ContainerDied","Data":"d72e278259d8c2c34bb186146085122a66e79ed90ea1e9a95f912ce3c1a41e07"} Oct 30 05:49:08.948261 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:08.948232 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 05:49:08.948458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:08.948284 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 05:49:09.718188 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.718159 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:09.796233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.796154 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca\") pod \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " Oct 30 05:49:09.796399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.796303 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk\" (UniqueName: \"kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk\") pod \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " Oct 30 05:49:09.796399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.796334 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca\") pod \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\" (UID: \"93e3e226-ff88-4e30-a0d5-2e1d3968092a\") " Oct 30 05:49:09.797220 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.797184 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca" pod "93e3e226-ff88-4e30-a0d5-2e1d3968092a" (UID: "93e3e226-ff88-4e30-a0d5-2e1d3968092a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:09.798953 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.798920 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk" pod "93e3e226-ff88-4e30-a0d5-2e1d3968092a" (UID: "93e3e226-ff88-4e30-a0d5-2e1d3968092a"). InnerVolumeSpecName "kube-api-access-9r9vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:09.801200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.801163 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca" pod "93e3e226-ff88-4e30-a0d5-2e1d3968092a" (UID: "93e3e226-ff88-4e30-a0d5-2e1d3968092a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:09.898649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.898617 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:09.898649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.898650 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r9vk\" (UniqueName: \"kubernetes.io/projected/93e3e226-ff88-4e30-a0d5-2e1d3968092a-kube-api-access-9r9vk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:09.898835 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.898665 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93e3e226-ff88-4e30-a0d5-2e1d3968092a-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:09.968603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.968567 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:09.970767 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.970024 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:09.971642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:09.971622 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 05:49:10.548472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:10.548443 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" Oct 30 05:49:10.548647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:10.548477 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc" event={"ID":"93e3e226-ff88-4e30-a0d5-2e1d3968092a","Type":"ContainerDied","Data":"9cdfb6fecc834beba865d5fa766762e7499f8b7a69d24a942e580b7a2972603c"} Oct 30 05:49:10.548647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:10.548524 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cdfb6fecc834beba865d5fa766762e7499f8b7a69d24a942e580b7a2972603c" Oct 30 05:49:11.971575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:11.971540 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv"] Oct 30 05:49:11.972160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:11.972146 5565 cpu_manager.go:414] "containerMap: removing container" podUID="93e3e226-ff88-4e30-a0d5-2e1d3968092a" containerName="revoke-keys" Oct 30 05:49:11.972209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:11.972161 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93e3e226-ff88-4e30-a0d5-2e1d3968092a" containerName="revoke-keys" Oct 30 05:49:11.972266 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:11.972256 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="93e3e226-ff88-4e30-a0d5-2e1d3968092a" containerName="revoke-keys" Oct 30 05:49:11.975713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:11.975697 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.005095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.005069 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:12.005273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.005251 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:12.005442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.005216 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:12.005442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.005126 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:12.012266 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.012241 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv"] Oct 30 05:49:12.133505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.133466 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.133680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.133663 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfhwv\" (UniqueName: \"kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.133729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.133692 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.235050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.234980 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.235180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.235162 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfhwv\" (UniqueName: \"kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.235222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.235202 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.235694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.235676 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.237544 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.237511 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.258384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.258356 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfhwv\" (UniqueName: \"kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv\") pod \"maas-api-revoke-sub-aa0cb2d9-6r7wv\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.289236 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.289217 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:12.601802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:12.601769 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv"] Oct 30 05:49:12.602168 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:12.602140 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15569a1c_0527_479d_9333_59be9455ff40.slice/crio-af4281d848f927a10cf37d2b700970f46603efd26b4ee91c97771d8bb3e49d55 WatchSource:0}: Error finding container af4281d848f927a10cf37d2b700970f46603efd26b4ee91c97771d8bb3e49d55: Status 404 returned error can't find the container with id af4281d848f927a10cf37d2b700970f46603efd26b4ee91c97771d8bb3e49d55 Oct 30 05:49:13.581760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:13.581721 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" event={"ID":"15569a1c-0527-479d-9333-59be9455ff40","Type":"ContainerStarted","Data":"6e548dda7ede91d9b8f21afb51dff47b28511b0487d8b38f4672a5912727757b"} Oct 30 05:49:13.582286 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:13.581788 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" event={"ID":"15569a1c-0527-479d-9333-59be9455ff40","Type":"ContainerStarted","Data":"af4281d848f927a10cf37d2b700970f46603efd26b4ee91c97771d8bb3e49d55"} Oct 30 05:49:13.626071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:13.626021 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" podStartSLOduration=2.626001576 podStartE2EDuration="2.626001576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:13.625275532 +0000 UTC m=+1060.200962585" watchObservedRunningTime="2026-10-30 05:49:13.626001576 +0000 UTC m=+1060.201688630" Oct 30 05:49:14.635552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:14.635266 5565 generic.go:356] "Generic (PLEG): container finished" podID="15569a1c-0527-479d-9333-59be9455ff40" containerID="6e548dda7ede91d9b8f21afb51dff47b28511b0487d8b38f4672a5912727757b" exitCode=0 Oct 30 05:49:14.635552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:14.635364 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" event={"ID":"15569a1c-0527-479d-9333-59be9455ff40","Type":"ContainerDied","Data":"6e548dda7ede91d9b8f21afb51dff47b28511b0487d8b38f4672a5912727757b"} Oct 30 05:49:15.299551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.299213 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.306551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.303917 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.310549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.307727 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:49:15.523551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.522627 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.581614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.580702 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:49:15.606552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.605629 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.666553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.663656 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.782746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.782624 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.782955 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.782792 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.795557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.784590 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.836612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.835780 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.908554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.906650 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:15.964553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:15.962644 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.188734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.188622 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.306662 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.306613 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.307632 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.307608 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:49:16.309432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.307853 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.309910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.309893 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:49:16.639669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.637207 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:49:16.639669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.637867 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:49:16.654551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.648311 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.654551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.648675 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:49:16.705285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.686604 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.705285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.686938 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:49:16.705285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.688607 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded" Oct 30 05:49:16.705285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.689167 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:49:16.815053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.789027 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.815053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.790358 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:49:16.815053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.813948 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:49:16.815053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.814166 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 05:49:16.815053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.814418 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:49:16.815053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.814475 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:49:16.837738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.836781 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:49:16.837738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.837158 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:49:16.851569 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.847640 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": context deadline exceeded" Oct 30 05:49:16.969468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.908128 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.969468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.931664 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:49:16.969468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.962868 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:16.969468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:16.965070 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:49:17.458611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.400879 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 05:49:17.458611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.401202 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:49:17.648858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.647519 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.648858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.648591 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:49:17.866416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.697982 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.914652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.873639 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.914652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.879654 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.914652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.896771 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:49:17.915381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.915048 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:17.932711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.916452 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.022552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.985649 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": context deadline exceeded" Oct 30 05:49:18.022552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.985771 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:49:18.022552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.986116 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 05:49:18.022552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:17.986436 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded" Oct 30 05:49:18.060576 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.057556 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.588863 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:49:18.588824 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:49:18.799431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.610651 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 05:49:18.983521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.983440 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:18.983521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.649118 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:49:18.984093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.649156 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.984093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.984000 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:49:18.984093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.941960 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.984596 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.957601 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:49:18.984788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.984772 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:49:18.985368 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.985350 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:49:18.986407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.957464 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.987029 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.610444 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.988002 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.987977 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:18.990836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:18.988340 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:19.003183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.003159 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" event={"ID":"15569a1c-0527-479d-9333-59be9455ff40","Type":"ContainerDied","Data":"af4281d848f927a10cf37d2b700970f46603efd26b4ee91c97771d8bb3e49d55"} Oct 30 05:49:19.003353 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.003334 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af4281d848f927a10cf37d2b700970f46603efd26b4ee91c97771d8bb3e49d55" Oct 30 05:49:19.006124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.005619 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:49:19.008122 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.007632 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:49:19.009148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.008649 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:49:19.014291 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.014271 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:49:19.015174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.015155 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:49:19.020774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.020744 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:49:19.021774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.021757 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:49:19.022320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.022305 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:49:19.025943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.025927 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:49:19.093944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.093916 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca\") pod \"15569a1c-0527-479d-9333-59be9455ff40\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " Oct 30 05:49:19.094113 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.094000 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv\" (UniqueName: \"kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv\") pod \"15569a1c-0527-479d-9333-59be9455ff40\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " Oct 30 05:49:19.094183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.094137 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca\") pod \"15569a1c-0527-479d-9333-59be9455ff40\" (UID: \"15569a1c-0527-479d-9333-59be9455ff40\") " Oct 30 05:49:19.094317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.094290 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca" pod "15569a1c-0527-479d-9333-59be9455ff40" (UID: "15569a1c-0527-479d-9333-59be9455ff40"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:19.095268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.095243 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/15569a1c-0527-479d-9333-59be9455ff40-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:19.096486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.096460 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv" pod "15569a1c-0527-479d-9333-59be9455ff40" (UID: "15569a1c-0527-479d-9333-59be9455ff40"). InnerVolumeSpecName "kube-api-access-zfhwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:19.097485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.097459 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca" pod "15569a1c-0527-479d-9333-59be9455ff40" (UID: "15569a1c-0527-479d-9333-59be9455ff40"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:19.190448 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.190420 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl"] Oct 30 05:49:19.191363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.191342 5565 cpu_manager.go:414] "containerMap: removing container" podUID="15569a1c-0527-479d-9333-59be9455ff40" containerName="revoke-keys" Oct 30 05:49:19.191423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.191367 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15569a1c-0527-479d-9333-59be9455ff40" containerName="revoke-keys" Oct 30 05:49:19.191616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.191603 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="15569a1c-0527-479d-9333-59be9455ff40" containerName="revoke-keys" Oct 30 05:49:19.195904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.195880 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.196983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.196957 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfhwv\" (UniqueName: \"kubernetes.io/projected/15569a1c-0527-479d-9333-59be9455ff40-kube-api-access-zfhwv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:19.196983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.196981 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/15569a1c-0527-479d-9333-59be9455ff40-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:19.221734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.221710 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl"] Oct 30 05:49:19.298295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.298265 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.298485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.298421 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.298633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.298615 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lznw5\" (UniqueName: \"kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.399711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.399680 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lznw5\" (UniqueName: \"kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.399869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.399816 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.399934 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.399884 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.400505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.400480 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.402260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.402226 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.427748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.427724 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lznw5\" (UniqueName: \"kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5\") pod \"maas-api-revoke-sub-e2c4e33c-tnzbl\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.513577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.513509 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:19.845152 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:19.845109 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29cee4e_2980_4266_997b_f017c448100b.slice/crio-d69958f848a51ab04c68f40dd052ea945765517975fa70fbf6904fb565e5c432 WatchSource:0}: Error finding container d69958f848a51ab04c68f40dd052ea945765517975fa70fbf6904fb565e5c432: Status 404 returned error can't find the container with id d69958f848a51ab04c68f40dd052ea945765517975fa70fbf6904fb565e5c432 Oct 30 05:49:19.845475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:19.845457 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl"] Oct 30 05:49:20.014276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:20.014245 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" event={"ID":"a29cee4e-2980-4266-997b-f017c448100b","Type":"ContainerStarted","Data":"d69958f848a51ab04c68f40dd052ea945765517975fa70fbf6904fb565e5c432"} Oct 30 05:49:20.014276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:20.014269 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv" Oct 30 05:49:21.024244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.024216 5565 generic.go:356] "Generic (PLEG): container finished" podID="a29cee4e-2980-4266-997b-f017c448100b" containerID="8f20b5a5d1ca34c53aca3a61d4441bf61b23cde004611fef58760ff5479d2e35" exitCode=0 Oct 30 05:49:21.024606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.024291 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" event={"ID":"a29cee4e-2980-4266-997b-f017c448100b","Type":"ContainerDied","Data":"8f20b5a5d1ca34c53aca3a61d4441bf61b23cde004611fef58760ff5479d2e35"} Oct 30 05:49:21.744071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.744037 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8"] Oct 30 05:49:21.748384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.748363 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.759849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.759042 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8"] Oct 30 05:49:21.832065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.832028 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.832258 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.832145 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.832258 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.832217 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl4nt\" (UniqueName: \"kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.933357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.933326 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.933519 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.933398 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.933519 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.933450 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl4nt\" (UniqueName: \"kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.934090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.934066 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.935724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.935704 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:21.963595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:21.963571 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl4nt\" (UniqueName: \"kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt\") pod \"maas-api-revoke-sub-5d67624b-6ndt8\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:22.066798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.066768 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:22.188823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.188793 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:22.238343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.238312 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca\") pod \"a29cee4e-2980-4266-997b-f017c448100b\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " Oct 30 05:49:22.238496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.238471 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca\") pod \"a29cee4e-2980-4266-997b-f017c448100b\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " Oct 30 05:49:22.238711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.238689 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5\" (UniqueName: \"kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5\") pod \"a29cee4e-2980-4266-997b-f017c448100b\" (UID: \"a29cee4e-2980-4266-997b-f017c448100b\") " Oct 30 05:49:22.238782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.238728 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca" pod "a29cee4e-2980-4266-997b-f017c448100b" (UID: "a29cee4e-2980-4266-997b-f017c448100b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:22.239307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.239282 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29cee4e-2980-4266-997b-f017c448100b-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:22.241355 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.241326 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5" pod "a29cee4e-2980-4266-997b-f017c448100b" (UID: "a29cee4e-2980-4266-997b-f017c448100b"). InnerVolumeSpecName "kube-api-access-lznw5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:22.241751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.241719 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca" pod "a29cee4e-2980-4266-997b-f017c448100b" (UID: "a29cee4e-2980-4266-997b-f017c448100b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:22.341017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.340947 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29cee4e-2980-4266-997b-f017c448100b-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:22.341017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.340975 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lznw5\" (UniqueName: \"kubernetes.io/projected/a29cee4e-2980-4266-997b-f017c448100b-kube-api-access-lznw5\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:22.388563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:22.385295 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3260ead5_1f24_4f40_a728_b152c725fb36.slice/crio-a206178f9dfe12f0c79d7a8c6dc2932ccfd592bd26d06921c545d2473e3a5907 WatchSource:0}: Error finding container a206178f9dfe12f0c79d7a8c6dc2932ccfd592bd26d06921c545d2473e3a5907: Status 404 returned error can't find the container with id a206178f9dfe12f0c79d7a8c6dc2932ccfd592bd26d06921c545d2473e3a5907 Oct 30 05:49:22.388563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:22.386045 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8"] Oct 30 05:49:23.045737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:23.045702 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" event={"ID":"3260ead5-1f24-4f40-a728-b152c725fb36","Type":"ContainerStarted","Data":"77b657c107609ce9b639f7a80b1c28a76aa9455f723979934bf9cca01a161ced"} Oct 30 05:49:23.046005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:23.045972 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" event={"ID":"3260ead5-1f24-4f40-a728-b152c725fb36","Type":"ContainerStarted","Data":"a206178f9dfe12f0c79d7a8c6dc2932ccfd592bd26d06921c545d2473e3a5907"} Oct 30 05:49:23.047414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:23.047396 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" Oct 30 05:49:23.047560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:23.047441 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl" event={"ID":"a29cee4e-2980-4266-997b-f017c448100b","Type":"ContainerDied","Data":"d69958f848a51ab04c68f40dd052ea945765517975fa70fbf6904fb565e5c432"} Oct 30 05:49:23.047560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:23.047493 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d69958f848a51ab04c68f40dd052ea945765517975fa70fbf6904fb565e5c432" Oct 30 05:49:23.091927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:23.091877 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" podStartSLOduration=2.091862976 podStartE2EDuration="2.091862976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:23.087662729 +0000 UTC m=+1069.663349785" watchObservedRunningTime="2026-10-30 05:49:23.091862976 +0000 UTC m=+1069.667550030" Oct 30 05:49:24.058814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:24.058780 5565 generic.go:356] "Generic (PLEG): container finished" podID="3260ead5-1f24-4f40-a728-b152c725fb36" containerID="77b657c107609ce9b639f7a80b1c28a76aa9455f723979934bf9cca01a161ced" exitCode=0 Oct 30 05:49:24.058991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:24.058877 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" event={"ID":"3260ead5-1f24-4f40-a728-b152c725fb36","Type":"ContainerDied","Data":"77b657c107609ce9b639f7a80b1c28a76aa9455f723979934bf9cca01a161ced"} Oct 30 05:49:25.288517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.288482 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:25.426732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.426642 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca\") pod \"3260ead5-1f24-4f40-a728-b152c725fb36\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " Oct 30 05:49:25.426732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.426709 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca\") pod \"3260ead5-1f24-4f40-a728-b152c725fb36\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " Oct 30 05:49:25.426939 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.426764 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt\" (UniqueName: \"kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt\") pod \"3260ead5-1f24-4f40-a728-b152c725fb36\" (UID: \"3260ead5-1f24-4f40-a728-b152c725fb36\") " Oct 30 05:49:25.427133 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.427107 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca" pod "3260ead5-1f24-4f40-a728-b152c725fb36" (UID: "3260ead5-1f24-4f40-a728-b152c725fb36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:25.427450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.427426 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:25.428938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.428899 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt" pod "3260ead5-1f24-4f40-a728-b152c725fb36" (UID: "3260ead5-1f24-4f40-a728-b152c725fb36"). InnerVolumeSpecName "kube-api-access-sl4nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:25.429028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.428997 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca" pod "3260ead5-1f24-4f40-a728-b152c725fb36" (UID: "3260ead5-1f24-4f40-a728-b152c725fb36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:25.529696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.529649 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3260ead5-1f24-4f40-a728-b152c725fb36-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:25.529696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:25.529693 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl4nt\" (UniqueName: \"kubernetes.io/projected/3260ead5-1f24-4f40-a728-b152c725fb36-kube-api-access-sl4nt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:26.101126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:26.101096 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" Oct 30 05:49:26.101303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:26.101127 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8" event={"ID":"3260ead5-1f24-4f40-a728-b152c725fb36","Type":"ContainerDied","Data":"a206178f9dfe12f0c79d7a8c6dc2932ccfd592bd26d06921c545d2473e3a5907"} Oct 30 05:49:26.101303 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:26.101183 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a206178f9dfe12f0c79d7a8c6dc2932ccfd592bd26d06921c545d2473e3a5907" Oct 30 05:49:28.101696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.101657 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr"] Oct 30 05:49:28.102739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.102718 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a29cee4e-2980-4266-997b-f017c448100b" containerName="revoke-keys" Oct 30 05:49:28.102800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.102742 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29cee4e-2980-4266-997b-f017c448100b" containerName="revoke-keys" Oct 30 05:49:28.102800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.102765 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3260ead5-1f24-4f40-a728-b152c725fb36" containerName="revoke-keys" Oct 30 05:49:28.102800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.102774 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3260ead5-1f24-4f40-a728-b152c725fb36" containerName="revoke-keys" Oct 30 05:49:28.102943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.102930 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29cee4e-2980-4266-997b-f017c448100b" containerName="revoke-keys" Oct 30 05:49:28.102979 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.102950 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3260ead5-1f24-4f40-a728-b152c725fb36" containerName="revoke-keys" Oct 30 05:49:28.108601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.108577 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.134200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.134172 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr"] Oct 30 05:49:28.134419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.134244 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:28.134552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.134403 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:28.143435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.140698 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:28.143435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.140738 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:28.274111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.274077 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfjwq\" (UniqueName: \"kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.274275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.274198 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.274275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.274260 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.375378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.375305 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfjwq\" (UniqueName: \"kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.375378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.375368 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.375563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.375411 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.375996 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.375971 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.377752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.377730 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.399279 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.399248 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfjwq\" (UniqueName: \"kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq\") pod \"maas-api-revoke-sub-88a7c916-7vzxr\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.445216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.445196 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:28.758141 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:28.758107 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf1001f_23fd_4f6d_ba7d_e5b80d3198dd.slice/crio-4020b92749de8b0ed59527d672942623dcd8771c2a891c9a76d26fa756a72992 WatchSource:0}: Error finding container 4020b92749de8b0ed59527d672942623dcd8771c2a891c9a76d26fa756a72992: Status 404 returned error can't find the container with id 4020b92749de8b0ed59527d672942623dcd8771c2a891c9a76d26fa756a72992 Oct 30 05:49:28.758774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:28.758748 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr"] Oct 30 05:49:29.141436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.141397 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" event={"ID":"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd","Type":"ContainerStarted","Data":"6552b12d22d4cc056375db924815dec572b44dc35cd6e01e16103ea2813ac2b6"} Oct 30 05:49:29.141867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.141460 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" event={"ID":"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd","Type":"ContainerStarted","Data":"4020b92749de8b0ed59527d672942623dcd8771c2a891c9a76d26fa756a72992"} Oct 30 05:49:29.184578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.184508 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" podStartSLOduration=1.184494845 podStartE2EDuration="1.184494845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:29.182644771 +0000 UTC m=+1075.758331826" watchObservedRunningTime="2026-10-30 05:49:29.184494845 +0000 UTC m=+1075.760181929" Oct 30 05:49:29.558359 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.558326 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k"] Oct 30 05:49:29.563423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.563396 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.578899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.578872 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 05:49:29.578899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.578874 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 05:49:29.579097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.578904 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-h6vpc\"" Oct 30 05:49:29.579097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.578874 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 05:49:29.598024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.597996 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k"] Oct 30 05:49:29.692013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.691982 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.692147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.692019 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.692147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.692129 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.692238 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.692167 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.692291 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.692249 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.692328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.692294 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.692328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.692313 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r26dr\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.793972 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.794015 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.794186 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.794226 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.794268 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.794307 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.794385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.794322 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r26dr\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.795033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.795007 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.796418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.796388 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.796560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.796457 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.796789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.796772 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.796874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.796804 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.818804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.818744 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r26dr\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.823275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.823255 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-8s79k\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:29.879242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:29.879219 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:30.152577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:30.152482 5565 generic.go:356] "Generic (PLEG): container finished" podID="daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" containerID="6552b12d22d4cc056375db924815dec572b44dc35cd6e01e16103ea2813ac2b6" exitCode=0 Oct 30 05:49:30.152577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:30.152554 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" event={"ID":"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd","Type":"ContainerDied","Data":"6552b12d22d4cc056375db924815dec572b44dc35cd6e01e16103ea2813ac2b6"} Oct 30 05:49:30.216879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:30.216848 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k"] Oct 30 05:49:30.222520 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:30.222497 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78cd1982_682a_4f4b_9cba_3b7078cff5a2.slice/crio-fadcee142e03c2f9a5965fa1438f910ffe1f308dc21a438088531234098e18a2 WatchSource:0}: Error finding container fadcee142e03c2f9a5965fa1438f910ffe1f308dc21a438088531234098e18a2: Status 404 returned error can't find the container with id fadcee142e03c2f9a5965fa1438f910ffe1f308dc21a438088531234098e18a2 Oct 30 05:49:30.225118 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:30.225102 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 05:49:31.164429 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.164397 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" event={"ID":"78cd1982-682a-4f4b-9cba-3b7078cff5a2","Type":"ContainerStarted","Data":"fadcee142e03c2f9a5965fa1438f910ffe1f308dc21a438088531234098e18a2"} Oct 30 05:49:31.346345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.346318 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:31.521215 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.521182 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca\") pod \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " Oct 30 05:49:31.521393 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.521359 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq\" (UniqueName: \"kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq\") pod \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " Oct 30 05:49:31.521469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.521420 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca\") pod \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\" (UID: \"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd\") " Oct 30 05:49:31.522102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.521962 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca" pod "daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" (UID: "daf1001f-23fd-4f6d-ba7d-e5b80d3198dd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:31.522102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.522075 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:31.523641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.523610 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq" pod "daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" (UID: "daf1001f-23fd-4f6d-ba7d-e5b80d3198dd"). InnerVolumeSpecName "kube-api-access-hfjwq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:31.523755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.523674 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca" pod "daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" (UID: "daf1001f-23fd-4f6d-ba7d-e5b80d3198dd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:31.623222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.623192 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfjwq\" (UniqueName: \"kubernetes.io/projected/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-kube-api-access-hfjwq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:31.623222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:31.623218 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:32.175460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:32.175429 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" Oct 30 05:49:32.175460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:32.175447 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr" event={"ID":"daf1001f-23fd-4f6d-ba7d-e5b80d3198dd","Type":"ContainerDied","Data":"4020b92749de8b0ed59527d672942623dcd8771c2a891c9a76d26fa756a72992"} Oct 30 05:49:32.176056 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:32.175568 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4020b92749de8b0ed59527d672942623dcd8771c2a891c9a76d26fa756a72992" Oct 30 05:49:32.687870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:32.687839 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:32.687967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:32.687902 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:49:33.189566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:33.189295 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" event={"ID":"78cd1982-682a-4f4b-9cba-3b7078cff5a2","Type":"ContainerStarted","Data":"c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00"} Oct 30 05:49:33.189566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:33.189363 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:33.192357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:33.192327 5565 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8s79k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 05:49:33.192481 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:33.192384 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 05:49:33.256090 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:33.256024 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" podStartSLOduration=1.794807289 podStartE2EDuration="4.256005598s" totalImagesPullingTime="2.461198309s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:33.249177933 +0000 UTC m=+1079.824864987" watchObservedRunningTime="2026-10-30 05:49:33.256005598 +0000 UTC m=+1079.831692652" Oct 30 05:49:34.197390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:34.197362 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 05:49:35.317606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.317575 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7"] Oct 30 05:49:35.318487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.318462 5565 cpu_manager.go:414] "containerMap: removing container" podUID="daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" containerName="revoke-keys" Oct 30 05:49:35.318487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.318487 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" containerName="revoke-keys" Oct 30 05:49:35.318826 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.318801 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" containerName="revoke-keys" Oct 30 05:49:35.322962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.322941 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.345767 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.345742 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7"] Oct 30 05:49:35.351784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.351760 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:35.351967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.351948 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:35.352083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.351837 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:35.352083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.351796 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:35.382430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.382410 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccg4g\" (UniqueName: \"kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.382613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.382593 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.382680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.382662 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.483708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.483679 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.483876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.483751 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.483960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.483897 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccg4g\" (UniqueName: \"kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.484610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.484476 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.486508 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.486487 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.511624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.511603 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccg4g\" (UniqueName: \"kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g\") pod \"maas-api-revoke-sub-eab7e800-2qpn7\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.637103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.637022 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:35.956175 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:35.956133 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd49aa9be_9cab_427e_89bb_c347cd359102.slice/crio-8393472233e5143936722c1f110abd1a359f94d495a2dbd9ad7a57a746113197 WatchSource:0}: Error finding container 8393472233e5143936722c1f110abd1a359f94d495a2dbd9ad7a57a746113197: Status 404 returned error can't find the container with id 8393472233e5143936722c1f110abd1a359f94d495a2dbd9ad7a57a746113197 Oct 30 05:49:35.957920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:35.957780 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7"] Oct 30 05:49:36.213628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:36.213562 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" event={"ID":"d49aa9be-9cab-427e-89bb-c347cd359102","Type":"ContainerStarted","Data":"8393472233e5143936722c1f110abd1a359f94d495a2dbd9ad7a57a746113197"} Oct 30 05:49:37.224482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:37.224323 5565 generic.go:356] "Generic (PLEG): container finished" podID="d49aa9be-9cab-427e-89bb-c347cd359102" containerID="d72dc5acfdd75ae9d20d4918ac884e072b0c191a2ef5f8aafce7e522d3da2360" exitCode=0 Oct 30 05:49:37.224482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:37.224413 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" event={"ID":"d49aa9be-9cab-427e-89bb-c347cd359102","Type":"ContainerDied","Data":"d72dc5acfdd75ae9d20d4918ac884e072b0c191a2ef5f8aafce7e522d3da2360"} Oct 30 05:49:38.408870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.408841 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:38.521151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.521122 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca\") pod \"d49aa9be-9cab-427e-89bb-c347cd359102\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " Oct 30 05:49:38.521359 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.521339 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca\") pod \"d49aa9be-9cab-427e-89bb-c347cd359102\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " Oct 30 05:49:38.521469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.521444 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca" pod "d49aa9be-9cab-427e-89bb-c347cd359102" (UID: "d49aa9be-9cab-427e-89bb-c347cd359102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:38.521564 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.521452 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g\" (UniqueName: \"kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g\") pod \"d49aa9be-9cab-427e-89bb-c347cd359102\" (UID: \"d49aa9be-9cab-427e-89bb-c347cd359102\") " Oct 30 05:49:38.521960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.521943 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:38.523328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.523310 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca" pod "d49aa9be-9cab-427e-89bb-c347cd359102" (UID: "d49aa9be-9cab-427e-89bb-c347cd359102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:38.523476 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.523456 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g" pod "d49aa9be-9cab-427e-89bb-c347cd359102" (UID: "d49aa9be-9cab-427e-89bb-c347cd359102"). InnerVolumeSpecName "kube-api-access-ccg4g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:38.623914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.623891 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccg4g\" (UniqueName: \"kubernetes.io/projected/d49aa9be-9cab-427e-89bb-c347cd359102-kube-api-access-ccg4g\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:38.624025 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:38.623916 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d49aa9be-9cab-427e-89bb-c347cd359102-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:39.240475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:39.240445 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" Oct 30 05:49:39.240673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:39.240483 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7" event={"ID":"d49aa9be-9cab-427e-89bb-c347cd359102","Type":"ContainerDied","Data":"8393472233e5143936722c1f110abd1a359f94d495a2dbd9ad7a57a746113197"} Oct 30 05:49:39.240673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:39.240571 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8393472233e5143936722c1f110abd1a359f94d495a2dbd9ad7a57a746113197" Oct 30 05:49:41.362993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.362962 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2"] Oct 30 05:49:41.363620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.363604 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d49aa9be-9cab-427e-89bb-c347cd359102" containerName="revoke-keys" Oct 30 05:49:41.363620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.363620 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d49aa9be-9cab-427e-89bb-c347cd359102" containerName="revoke-keys" Oct 30 05:49:41.363727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.363721 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d49aa9be-9cab-427e-89bb-c347cd359102" containerName="revoke-keys" Oct 30 05:49:41.367744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.367727 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.389164 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.389142 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:41.389297 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.389281 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:41.389343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.389285 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:41.402475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.402449 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:41.409270 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.409242 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2"] Oct 30 05:49:41.471733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.471706 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.471921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.471901 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.472017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.472003 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6w4xq\" (UniqueName: \"kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.573006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.572982 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.573127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.573045 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6w4xq\" (UniqueName: \"kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.573127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.573101 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.573790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.573762 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.575276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.575259 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.597008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.596985 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6w4xq\" (UniqueName: \"kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq\") pod \"maas-api-revoke-sub-97f3dd70-2xvn2\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.681955 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.681901 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:41.994546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:41.994452 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2"] Oct 30 05:49:42.005417 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:42.005384 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc8d83c9_c5ad_40a9_8953_189df1ec4027.slice/crio-8f6d230b05d38758faa9c3caea23ccc1f998c414a0e0f2ff4e4568ad22df68b2 WatchSource:0}: Error finding container 8f6d230b05d38758faa9c3caea23ccc1f998c414a0e0f2ff4e4568ad22df68b2: Status 404 returned error can't find the container with id 8f6d230b05d38758faa9c3caea23ccc1f998c414a0e0f2ff4e4568ad22df68b2 Oct 30 05:49:42.013622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:42.010827 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:49:42.265395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:42.265348 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" event={"ID":"dc8d83c9-c5ad-40a9-8953-189df1ec4027","Type":"ContainerStarted","Data":"8f6d230b05d38758faa9c3caea23ccc1f998c414a0e0f2ff4e4568ad22df68b2"} Oct 30 05:49:43.277063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:43.277021 5565 generic.go:356] "Generic (PLEG): container finished" podID="dc8d83c9-c5ad-40a9-8953-189df1ec4027" containerID="e7c0710edf9d827f1b7e2b09c0b8b1c01b2280e753395651e02d4b43582488e2" exitCode=0 Oct 30 05:49:43.277424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:43.277075 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" event={"ID":"dc8d83c9-c5ad-40a9-8953-189df1ec4027","Type":"ContainerDied","Data":"e7c0710edf9d827f1b7e2b09c0b8b1c01b2280e753395651e02d4b43582488e2"} Oct 30 05:49:44.693629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:44.692854 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:44.787552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:44.781828 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:44.841577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:44.841513 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:44.950618 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:44.938323 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded" Oct 30 05:49:45.003854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:44.991849 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:45.145232 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.129051 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:45.226603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.224683 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca\") pod \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " Oct 30 05:49:45.226603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.224784 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca\") pod \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " Oct 30 05:49:45.226603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.225037 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq\" (UniqueName: \"kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq\") pod \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\" (UID: \"dc8d83c9-c5ad-40a9-8953-189df1ec4027\") " Oct 30 05:49:45.255373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.255337 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca" pod "dc8d83c9-c5ad-40a9-8953-189df1ec4027" (UID: "dc8d83c9-c5ad-40a9-8953-189df1ec4027"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:45.297513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.297485 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:45.328611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.327692 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq" pod "dc8d83c9-c5ad-40a9-8953-189df1ec4027" (UID: "dc8d83c9-c5ad-40a9-8953-189df1ec4027"). InnerVolumeSpecName "kube-api-access-6w4xq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:45.328611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.327783 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca" pod "dc8d83c9-c5ad-40a9-8953-189df1ec4027" (UID: "dc8d83c9-c5ad-40a9-8953-189df1ec4027"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:45.343010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.340240 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" event={"ID":"dc8d83c9-c5ad-40a9-8953-189df1ec4027","Type":"ContainerDied","Data":"8f6d230b05d38758faa9c3caea23ccc1f998c414a0e0f2ff4e4568ad22df68b2"} Oct 30 05:49:45.343010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.340309 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f6d230b05d38758faa9c3caea23ccc1f998c414a0e0f2ff4e4568ad22df68b2" Oct 30 05:49:45.343010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.340517 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2" Oct 30 05:49:45.407548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.400767 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6w4xq\" (UniqueName: \"kubernetes.io/projected/dc8d83c9-c5ad-40a9-8953-189df1ec4027-kube-api-access-6w4xq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:45.407548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.400803 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dc8d83c9-c5ad-40a9-8953-189df1ec4027-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:45.448839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.431622 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:45.699552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.689649 5565 patch_prober.go:29] interesting pod/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:49:45.699552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.689716 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:45.699552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.694257 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:45.699552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.694684 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:49:45.848548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.847953 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:49:45.859548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.848806 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:49:45.949551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:45.947625 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:49:46.185669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.185507 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:49:46.291391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.291360 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:49:46.445954 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.445871 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk"] Oct 30 05:49:46.446747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.446723 5565 cpu_manager.go:414] "containerMap: removing container" podUID="dc8d83c9-c5ad-40a9-8953-189df1ec4027" containerName="revoke-keys" Oct 30 05:49:46.446747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.446744 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc8d83c9-c5ad-40a9-8953-189df1ec4027" containerName="revoke-keys" Oct 30 05:49:46.446940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.446886 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc8d83c9-c5ad-40a9-8953-189df1ec4027" containerName="revoke-keys" Oct 30 05:49:46.451061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.451041 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.468328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.468301 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:46.469029 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.469006 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:46.469193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.469080 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:46.469193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.469166 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:46.471728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.471705 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk"] Oct 30 05:49:46.577860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.577826 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.578071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.578054 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tr5z\" (UniqueName: \"kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.578148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.578136 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.679645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.679615 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tr5z\" (UniqueName: \"kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.679813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.679715 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.679813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.679781 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.680634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.680609 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.682469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.682442 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.703487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.703434 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tr5z\" (UniqueName: \"kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z\") pod \"maas-api-revoke-sub-30c12092-tpmmk\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:46.769102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:46.769079 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:47.098599 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:47.098558 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fa5a0ec_e74f_4085_af8f_37fb28865655.slice/crio-a5fd9d575f07ed6dac000800b1059803ba96f7252ef1bfdaeb1e6bbca8a443be WatchSource:0}: Error finding container a5fd9d575f07ed6dac000800b1059803ba96f7252ef1bfdaeb1e6bbca8a443be: Status 404 returned error can't find the container with id a5fd9d575f07ed6dac000800b1059803ba96f7252ef1bfdaeb1e6bbca8a443be Oct 30 05:49:47.100859 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.100834 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk"] Oct 30 05:49:47.377659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.377632 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" event={"ID":"7fa5a0ec-e74f-4085-af8f-37fb28865655","Type":"ContainerStarted","Data":"3cf6ac0160396ae7b46fe107e4ad2c5fb06710336d34b171d2610a4944c71570"} Oct 30 05:49:47.377789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.377680 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" event={"ID":"7fa5a0ec-e74f-4085-af8f-37fb28865655","Type":"ContainerStarted","Data":"a5fd9d575f07ed6dac000800b1059803ba96f7252ef1bfdaeb1e6bbca8a443be"} Oct 30 05:49:47.400201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.400176 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz"] Oct 30 05:49:47.404739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.404721 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.426026 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.426002 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz"] Oct 30 05:49:47.430880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.430824 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" podStartSLOduration=1.430808298 podStartE2EDuration="1.430808298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:47.422913132 +0000 UTC m=+1093.998600186" watchObservedRunningTime="2026-10-30 05:49:47.430808298 +0000 UTC m=+1094.006495352" Oct 30 05:49:47.489247 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.489217 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.489435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.489415 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.489559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.489542 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slgvk\" (UniqueName: \"kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.532822 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:49:47.532779 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:49:47.590973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.590810 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.591544 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.591173 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slgvk\" (UniqueName: \"kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.591544 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.591258 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.592023 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.591996 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.593215 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.593192 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.615715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.615659 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-slgvk\" (UniqueName: \"kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk\") pod \"maas-api-revoke-sub-f72470f6-7g2cz\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:47.718201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:47.718166 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:48.029522 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:48.029496 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz"] Oct 30 05:49:48.034563 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:48.034498 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb1675da_0614_4a54_bf5b_cd387988f3f8.slice/crio-9b8498c278cb56b637001a4a2e67f6db832c55f06de3839e5acff833add1766a WatchSource:0}: Error finding container 9b8498c278cb56b637001a4a2e67f6db832c55f06de3839e5acff833add1766a: Status 404 returned error can't find the container with id 9b8498c278cb56b637001a4a2e67f6db832c55f06de3839e5acff833add1766a Oct 30 05:49:48.387497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:48.387458 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" event={"ID":"fb1675da-0614-4a54-bf5b-cd387988f3f8","Type":"ContainerStarted","Data":"a880630fa4703a778ef623d88e91199be3783782bad2dcbc5bf41bab7c57f420"} Oct 30 05:49:48.387700 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:48.387517 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" event={"ID":"fb1675da-0614-4a54-bf5b-cd387988f3f8","Type":"ContainerStarted","Data":"9b8498c278cb56b637001a4a2e67f6db832c55f06de3839e5acff833add1766a"} Oct 30 05:49:48.389112 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:48.389091 5565 generic.go:356] "Generic (PLEG): container finished" podID="7fa5a0ec-e74f-4085-af8f-37fb28865655" containerID="3cf6ac0160396ae7b46fe107e4ad2c5fb06710336d34b171d2610a4944c71570" exitCode=0 Oct 30 05:49:48.389221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:48.389176 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" event={"ID":"7fa5a0ec-e74f-4085-af8f-37fb28865655","Type":"ContainerDied","Data":"3cf6ac0160396ae7b46fe107e4ad2c5fb06710336d34b171d2610a4944c71570"} Oct 30 05:49:48.430235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:48.430182 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" podStartSLOduration=1.430163993 podStartE2EDuration="1.430163993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:49:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:49:48.428416469 +0000 UTC m=+1095.004103537" watchObservedRunningTime="2026-10-30 05:49:48.430163993 +0000 UTC m=+1095.005851053" Oct 30 05:49:49.398514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.398473 5565 generic.go:356] "Generic (PLEG): container finished" podID="fb1675da-0614-4a54-bf5b-cd387988f3f8" containerID="a880630fa4703a778ef623d88e91199be3783782bad2dcbc5bf41bab7c57f420" exitCode=0 Oct 30 05:49:49.398924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.398568 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" event={"ID":"fb1675da-0614-4a54-bf5b-cd387988f3f8","Type":"ContainerDied","Data":"a880630fa4703a778ef623d88e91199be3783782bad2dcbc5bf41bab7c57f420"} Oct 30 05:49:49.574378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.574352 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:49.621950 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.621916 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca\") pod \"7fa5a0ec-e74f-4085-af8f-37fb28865655\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " Oct 30 05:49:49.622119 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.622084 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z\" (UniqueName: \"kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z\") pod \"7fa5a0ec-e74f-4085-af8f-37fb28865655\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " Oct 30 05:49:49.622258 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.622241 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca\") pod \"7fa5a0ec-e74f-4085-af8f-37fb28865655\" (UID: \"7fa5a0ec-e74f-4085-af8f-37fb28865655\") " Oct 30 05:49:49.622317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.622268 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca" pod "7fa5a0ec-e74f-4085-af8f-37fb28865655" (UID: "7fa5a0ec-e74f-4085-af8f-37fb28865655"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:49.623098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.623072 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:49.624451 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.624429 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca" pod "7fa5a0ec-e74f-4085-af8f-37fb28865655" (UID: "7fa5a0ec-e74f-4085-af8f-37fb28865655"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:49.624642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.624580 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z" pod "7fa5a0ec-e74f-4085-af8f-37fb28865655" (UID: "7fa5a0ec-e74f-4085-af8f-37fb28865655"). InnerVolumeSpecName "kube-api-access-5tr5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:49.724661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.724507 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tr5z\" (UniqueName: \"kubernetes.io/projected/7fa5a0ec-e74f-4085-af8f-37fb28865655-kube-api-access-5tr5z\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:49.724661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:49.724551 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fa5a0ec-e74f-4085-af8f-37fb28865655-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:50.408362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.408333 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" Oct 30 05:49:50.408798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.408375 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk" event={"ID":"7fa5a0ec-e74f-4085-af8f-37fb28865655","Type":"ContainerDied","Data":"a5fd9d575f07ed6dac000800b1059803ba96f7252ef1bfdaeb1e6bbca8a443be"} Oct 30 05:49:50.408798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.408435 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5fd9d575f07ed6dac000800b1059803ba96f7252ef1bfdaeb1e6bbca8a443be" Oct 30 05:49:50.578147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.578107 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:50.635611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.635577 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca\") pod \"fb1675da-0614-4a54-bf5b-cd387988f3f8\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " Oct 30 05:49:50.635757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.635676 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk\" (UniqueName: \"kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk\") pod \"fb1675da-0614-4a54-bf5b-cd387988f3f8\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " Oct 30 05:49:50.635823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.635810 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca\") pod \"fb1675da-0614-4a54-bf5b-cd387988f3f8\" (UID: \"fb1675da-0614-4a54-bf5b-cd387988f3f8\") " Oct 30 05:49:50.636233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.636200 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca" pod "fb1675da-0614-4a54-bf5b-cd387988f3f8" (UID: "fb1675da-0614-4a54-bf5b-cd387988f3f8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:50.636667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.636644 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:50.637816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.637788 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca" pod "fb1675da-0614-4a54-bf5b-cd387988f3f8" (UID: "fb1675da-0614-4a54-bf5b-cd387988f3f8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:50.637974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.637953 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk" pod "fb1675da-0614-4a54-bf5b-cd387988f3f8" (UID: "fb1675da-0614-4a54-bf5b-cd387988f3f8"). InnerVolumeSpecName "kube-api-access-slgvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:50.738446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.738384 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fb1675da-0614-4a54-bf5b-cd387988f3f8-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:50.738446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:50.738411 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slgvk\" (UniqueName: \"kubernetes.io/projected/fb1675da-0614-4a54-bf5b-cd387988f3f8-kube-api-access-slgvk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:51.417353 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:51.417325 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" Oct 30 05:49:51.417739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:51.417366 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz" event={"ID":"fb1675da-0614-4a54-bf5b-cd387988f3f8","Type":"ContainerDied","Data":"9b8498c278cb56b637001a4a2e67f6db832c55f06de3839e5acff833add1766a"} Oct 30 05:49:51.417739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:51.417416 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b8498c278cb56b637001a4a2e67f6db832c55f06de3839e5acff833add1766a" Oct 30 05:49:54.633613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.633565 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc"] Oct 30 05:49:54.634410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.634387 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fb1675da-0614-4a54-bf5b-cd387988f3f8" containerName="revoke-keys" Oct 30 05:49:54.634410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.634411 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb1675da-0614-4a54-bf5b-cd387988f3f8" containerName="revoke-keys" Oct 30 05:49:54.634583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.634437 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7fa5a0ec-e74f-4085-af8f-37fb28865655" containerName="revoke-keys" Oct 30 05:49:54.634583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.634444 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fa5a0ec-e74f-4085-af8f-37fb28865655" containerName="revoke-keys" Oct 30 05:49:54.634699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.634669 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb1675da-0614-4a54-bf5b-cd387988f3f8" containerName="revoke-keys" Oct 30 05:49:54.634699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.634687 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fa5a0ec-e74f-4085-af8f-37fb28865655" containerName="revoke-keys" Oct 30 05:49:54.638555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.638520 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.660753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.660724 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc"] Oct 30 05:49:54.667039 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.667010 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:49:54.667169 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.667085 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:49:54.685492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.685465 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.685745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.685728 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.685801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.685770 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbnx7\" (UniqueName: \"kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.723075 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.723034 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:49:54.723193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.723176 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:49:54.787236 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.787205 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.787385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.787256 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbnx7\" (UniqueName: \"kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.787385 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.787337 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.788031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.788008 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.791902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.791878 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.811225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.811199 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbnx7\" (UniqueName: \"kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7\") pod \"maas-api-revoke-sub-ca4b71f7-v49pc\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:54.952422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:54.952356 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:55.286875 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:55.286824 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1dd564c_e34f_4595_93e7_7d58ce9d7102.slice/crio-aefa42bf556bc097cf617bbe3255b48bfd4804c7d1c2e30010a4732ae554dc69 WatchSource:0}: Error finding container aefa42bf556bc097cf617bbe3255b48bfd4804c7d1c2e30010a4732ae554dc69: Status 404 returned error can't find the container with id aefa42bf556bc097cf617bbe3255b48bfd4804c7d1c2e30010a4732ae554dc69 Oct 30 05:49:55.288752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:55.288730 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc"] Oct 30 05:49:55.450044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:55.449999 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" event={"ID":"a1dd564c-e34f-4595-93e7-7d58ce9d7102","Type":"ContainerStarted","Data":"aefa42bf556bc097cf617bbe3255b48bfd4804c7d1c2e30010a4732ae554dc69"} Oct 30 05:49:56.458645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:56.458568 5565 generic.go:356] "Generic (PLEG): container finished" podID="a1dd564c-e34f-4595-93e7-7d58ce9d7102" containerID="0c5de13434f1d2581d2f8892a8d43abd4273d8eb512f56d7b73775fe3173bc02" exitCode=0 Oct 30 05:49:56.459015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:56.458655 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" event={"ID":"a1dd564c-e34f-4595-93e7-7d58ce9d7102","Type":"ContainerDied","Data":"0c5de13434f1d2581d2f8892a8d43abd4273d8eb512f56d7b73775fe3173bc02"} Oct 30 05:49:57.648894 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.648867 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:57.730499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.730425 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca\") pod \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " Oct 30 05:49:57.730673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.730582 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca\") pod \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " Oct 30 05:49:57.730673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.730622 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7\" (UniqueName: \"kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7\") pod \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\" (UID: \"a1dd564c-e34f-4595-93e7-7d58ce9d7102\") " Oct 30 05:49:57.730914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.730851 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca" pod "a1dd564c-e34f-4595-93e7-7d58ce9d7102" (UID: "a1dd564c-e34f-4595-93e7-7d58ce9d7102"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:49:57.731292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.731269 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:57.732836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.732806 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca" pod "a1dd564c-e34f-4595-93e7-7d58ce9d7102" (UID: "a1dd564c-e34f-4595-93e7-7d58ce9d7102"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:49:57.732924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.732852 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7" pod "a1dd564c-e34f-4595-93e7-7d58ce9d7102" (UID: "a1dd564c-e34f-4595-93e7-7d58ce9d7102"). InnerVolumeSpecName "kube-api-access-nbnx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:49:57.832984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.832946 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1dd564c-e34f-4595-93e7-7d58ce9d7102-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:57.832984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.832977 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbnx7\" (UniqueName: \"kubernetes.io/projected/a1dd564c-e34f-4595-93e7-7d58ce9d7102-kube-api-access-nbnx7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:49:57.891915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.891883 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778"] Oct 30 05:49:57.892521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.892506 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a1dd564c-e34f-4595-93e7-7d58ce9d7102" containerName="revoke-keys" Oct 30 05:49:57.892521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.892522 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1dd564c-e34f-4595-93e7-7d58ce9d7102" containerName="revoke-keys" Oct 30 05:49:57.892696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.892683 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1dd564c-e34f-4595-93e7-7d58ce9d7102" containerName="revoke-keys" Oct 30 05:49:57.897611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.897587 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:57.918237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:57.918202 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778"] Oct 30 05:49:58.035435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.035403 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.035619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.035465 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.035619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.035592 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcdf7\" (UniqueName: \"kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.136688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.136649 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcdf7\" (UniqueName: \"kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.136982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.136964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.137067 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.137020 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.137637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.137606 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.139378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.139351 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.161155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.161125 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcdf7\" (UniqueName: \"kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7\") pod \"maas-api-revoke-sub-9187914a-56778\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.213433 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.213408 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:49:58.484836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.484757 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" Oct 30 05:49:58.484836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.484789 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc" event={"ID":"a1dd564c-e34f-4595-93e7-7d58ce9d7102","Type":"ContainerDied","Data":"aefa42bf556bc097cf617bbe3255b48bfd4804c7d1c2e30010a4732ae554dc69"} Oct 30 05:49:58.485006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.484844 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aefa42bf556bc097cf617bbe3255b48bfd4804c7d1c2e30010a4732ae554dc69" Oct 30 05:49:58.518798 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:49:58.518773 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53a67de_159d_4406_9138_1dd8c088fcab.slice/crio-25adf0404b1fa11c4dbc971684ee27e01d2068dc887d5120c7a4d1533d489111 WatchSource:0}: Error finding container 25adf0404b1fa11c4dbc971684ee27e01d2068dc887d5120c7a4d1533d489111: Status 404 returned error can't find the container with id 25adf0404b1fa11c4dbc971684ee27e01d2068dc887d5120c7a4d1533d489111 Oct 30 05:49:58.520503 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:58.520480 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778"] Oct 30 05:49:59.495001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:59.494969 5565 generic.go:356] "Generic (PLEG): container finished" podID="a53a67de-159d-4406-9138-1dd8c088fcab" containerID="d03a7adb43166500b02efe2675cca7ebf7576410076a3aaafd5f5fef7229cae0" exitCode=0 Oct 30 05:49:59.495418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:59.495055 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" event={"ID":"a53a67de-159d-4406-9138-1dd8c088fcab","Type":"ContainerDied","Data":"d03a7adb43166500b02efe2675cca7ebf7576410076a3aaafd5f5fef7229cae0"} Oct 30 05:49:59.495418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:49:59.495105 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" event={"ID":"a53a67de-159d-4406-9138-1dd8c088fcab","Type":"ContainerStarted","Data":"25adf0404b1fa11c4dbc971684ee27e01d2068dc887d5120c7a4d1533d489111"} Oct 30 05:50:00.649290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.649265 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:50:00.772216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.772143 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca\") pod \"a53a67de-159d-4406-9138-1dd8c088fcab\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " Oct 30 05:50:00.772216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.772191 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca\") pod \"a53a67de-159d-4406-9138-1dd8c088fcab\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " Oct 30 05:50:00.772416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.772218 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7\" (UniqueName: \"kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7\") pod \"a53a67de-159d-4406-9138-1dd8c088fcab\" (UID: \"a53a67de-159d-4406-9138-1dd8c088fcab\") " Oct 30 05:50:00.772579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.772551 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca" pod "a53a67de-159d-4406-9138-1dd8c088fcab" (UID: "a53a67de-159d-4406-9138-1dd8c088fcab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:00.772715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.772700 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:00.774173 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.774150 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca" pod "a53a67de-159d-4406-9138-1dd8c088fcab" (UID: "a53a67de-159d-4406-9138-1dd8c088fcab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:00.774319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.774296 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7" pod "a53a67de-159d-4406-9138-1dd8c088fcab" (UID: "a53a67de-159d-4406-9138-1dd8c088fcab"). InnerVolumeSpecName "kube-api-access-zcdf7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:00.874152 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.874127 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a67de-159d-4406-9138-1dd8c088fcab-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:00.874152 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:00.874150 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcdf7\" (UniqueName: \"kubernetes.io/projected/a53a67de-159d-4406-9138-1dd8c088fcab-kube-api-access-zcdf7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:01.468081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.468052 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp"] Oct 30 05:50:01.468762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.468745 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a53a67de-159d-4406-9138-1dd8c088fcab" containerName="revoke-keys" Oct 30 05:50:01.468812 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.468763 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53a67de-159d-4406-9138-1dd8c088fcab" containerName="revoke-keys" Oct 30 05:50:01.468902 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.468892 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53a67de-159d-4406-9138-1dd8c088fcab" containerName="revoke-keys" Oct 30 05:50:01.472805 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.472785 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.500861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.500831 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp"] Oct 30 05:50:01.517866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.517832 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" event={"ID":"a53a67de-159d-4406-9138-1dd8c088fcab","Type":"ContainerDied","Data":"25adf0404b1fa11c4dbc971684ee27e01d2068dc887d5120c7a4d1533d489111"} Oct 30 05:50:01.518067 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.517893 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="25adf0404b1fa11c4dbc971684ee27e01d2068dc887d5120c7a4d1533d489111" Oct 30 05:50:01.518067 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.517998 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778" Oct 30 05:50:01.584191 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.584159 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtpw\" (UniqueName: \"kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.584323 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.584196 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.584467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.584447 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.685698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.685672 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.686126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.685727 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtpw\" (UniqueName: \"kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.686126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.685752 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.686585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.686562 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.688017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.687999 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.710183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.710160 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtpw\" (UniqueName: \"kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw\") pod \"maas-api-revoke-sub-4db310dc-2kgbp\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:01.787666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:01.787640 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:02.100782 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:02.100743 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd53c8aad_c181_402c_9dcd_0d4302d35bd2.slice/crio-4964f4f2e96c747107030c5209ce75eaf2280880ca83aa8dfd3bdc9281f66784 WatchSource:0}: Error finding container 4964f4f2e96c747107030c5209ce75eaf2280880ca83aa8dfd3bdc9281f66784: Status 404 returned error can't find the container with id 4964f4f2e96c747107030c5209ce75eaf2280880ca83aa8dfd3bdc9281f66784 Oct 30 05:50:02.102721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:02.102689 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp"] Oct 30 05:50:02.526752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:02.526715 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" event={"ID":"d53c8aad-c181-402c-9dcd-0d4302d35bd2","Type":"ContainerStarted","Data":"f0d82eb5c5afa3a2eba5e5aae264a8786d9242f451d6b51ed6e17576fbefb5dc"} Oct 30 05:50:02.526945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:02.526774 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" event={"ID":"d53c8aad-c181-402c-9dcd-0d4302d35bd2","Type":"ContainerStarted","Data":"4964f4f2e96c747107030c5209ce75eaf2280880ca83aa8dfd3bdc9281f66784"} Oct 30 05:50:02.571254 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:02.571204 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" podStartSLOduration=1.571191084 podStartE2EDuration="1.571191084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:02.568237169 +0000 UTC m=+1109.143924225" watchObservedRunningTime="2026-10-30 05:50:02.571191084 +0000 UTC m=+1109.146878137" Oct 30 05:50:03.536727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:03.536693 5565 generic.go:356] "Generic (PLEG): container finished" podID="d53c8aad-c181-402c-9dcd-0d4302d35bd2" containerID="f0d82eb5c5afa3a2eba5e5aae264a8786d9242f451d6b51ed6e17576fbefb5dc" exitCode=0 Oct 30 05:50:03.537110 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:03.536776 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" event={"ID":"d53c8aad-c181-402c-9dcd-0d4302d35bd2","Type":"ContainerDied","Data":"f0d82eb5c5afa3a2eba5e5aae264a8786d9242f451d6b51ed6e17576fbefb5dc"} Oct 30 05:50:04.748635 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.748605 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:04.828043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.827965 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca\") pod \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " Oct 30 05:50:04.828196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.828048 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw\" (UniqueName: \"kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw\") pod \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " Oct 30 05:50:04.828196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.828137 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca\") pod \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\" (UID: \"d53c8aad-c181-402c-9dcd-0d4302d35bd2\") " Oct 30 05:50:04.828395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.828371 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca" pod "d53c8aad-c181-402c-9dcd-0d4302d35bd2" (UID: "d53c8aad-c181-402c-9dcd-0d4302d35bd2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:04.828847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.828821 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:04.830629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.830591 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca" pod "d53c8aad-c181-402c-9dcd-0d4302d35bd2" (UID: "d53c8aad-c181-402c-9dcd-0d4302d35bd2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:04.831908 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.831884 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw" pod "d53c8aad-c181-402c-9dcd-0d4302d35bd2" (UID: "d53c8aad-c181-402c-9dcd-0d4302d35bd2"). InnerVolumeSpecName "kube-api-access-crtpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:04.930084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.930059 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtpw\" (UniqueName: \"kubernetes.io/projected/d53c8aad-c181-402c-9dcd-0d4302d35bd2-kube-api-access-crtpw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:04.930084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:04.930083 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d53c8aad-c181-402c-9dcd-0d4302d35bd2-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:05.439161 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.439125 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m"] Oct 30 05:50:05.439913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.439891 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d53c8aad-c181-402c-9dcd-0d4302d35bd2" containerName="revoke-keys" Oct 30 05:50:05.439913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.439912 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d53c8aad-c181-402c-9dcd-0d4302d35bd2" containerName="revoke-keys" Oct 30 05:50:05.440157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.440142 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d53c8aad-c181-402c-9dcd-0d4302d35bd2" containerName="revoke-keys" Oct 30 05:50:05.444434 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.444414 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.469172 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.469106 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m"] Oct 30 05:50:05.545306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.545229 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.545594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.545562 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.545766 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.545697 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdwbf\" (UniqueName: \"kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.567595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.567562 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" Oct 30 05:50:05.569806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.569783 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp" event={"ID":"d53c8aad-c181-402c-9dcd-0d4302d35bd2","Type":"ContainerDied","Data":"4964f4f2e96c747107030c5209ce75eaf2280880ca83aa8dfd3bdc9281f66784"} Oct 30 05:50:05.569956 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.569823 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4964f4f2e96c747107030c5209ce75eaf2280880ca83aa8dfd3bdc9281f66784" Oct 30 05:50:05.647256 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.647223 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.647809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.647780 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.648014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.647986 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdwbf\" (UniqueName: \"kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.648107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.648051 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.650122 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.650099 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.683363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.683339 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdwbf\" (UniqueName: \"kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf\") pod \"maas-api-revoke-sub-426f7b1e-8pd8m\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:05.760321 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:05.760297 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:06.075929 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:06.075899 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf73b12d_1ea0_4542_ab67_69e662133fee.slice/crio-80abc3c5b53fd59f7c6d7f139b86d6b8f895950502a474a531b4cbc63dff3bc6 WatchSource:0}: Error finding container 80abc3c5b53fd59f7c6d7f139b86d6b8f895950502a474a531b4cbc63dff3bc6: Status 404 returned error can't find the container with id 80abc3c5b53fd59f7c6d7f139b86d6b8f895950502a474a531b4cbc63dff3bc6 Oct 30 05:50:06.077680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:06.077610 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m"] Oct 30 05:50:06.576989 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:06.576953 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" event={"ID":"cf73b12d-1ea0-4542-ab67-69e662133fee","Type":"ContainerStarted","Data":"5286633b865b3079a8791c3355d6fadc6d30165159fd486faa6405468625d8ae"} Oct 30 05:50:06.577135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:06.577008 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" event={"ID":"cf73b12d-1ea0-4542-ab67-69e662133fee","Type":"ContainerStarted","Data":"80abc3c5b53fd59f7c6d7f139b86d6b8f895950502a474a531b4cbc63dff3bc6"} Oct 30 05:50:06.629452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:06.629404 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" podStartSLOduration=1.629391502 podStartE2EDuration="1.629391502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:06.623267176 +0000 UTC m=+1113.198954229" watchObservedRunningTime="2026-10-30 05:50:06.629391502 +0000 UTC m=+1113.205078555" Oct 30 05:50:07.585344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:07.585313 5565 generic.go:356] "Generic (PLEG): container finished" podID="cf73b12d-1ea0-4542-ab67-69e662133fee" containerID="5286633b865b3079a8791c3355d6fadc6d30165159fd486faa6405468625d8ae" exitCode=0 Oct 30 05:50:07.585840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:07.585397 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" event={"ID":"cf73b12d-1ea0-4542-ab67-69e662133fee","Type":"ContainerDied","Data":"5286633b865b3079a8791c3355d6fadc6d30165159fd486faa6405468625d8ae"} Oct 30 05:50:08.748340 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.748312 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:08.897583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.897470 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf\" (UniqueName: \"kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf\") pod \"cf73b12d-1ea0-4542-ab67-69e662133fee\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " Oct 30 05:50:08.897896 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.897868 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca\") pod \"cf73b12d-1ea0-4542-ab67-69e662133fee\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " Oct 30 05:50:08.898024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.897972 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca\") pod \"cf73b12d-1ea0-4542-ab67-69e662133fee\" (UID: \"cf73b12d-1ea0-4542-ab67-69e662133fee\") " Oct 30 05:50:08.898688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.898657 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca" pod "cf73b12d-1ea0-4542-ab67-69e662133fee" (UID: "cf73b12d-1ea0-4542-ab67-69e662133fee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:08.900109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.900081 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf" pod "cf73b12d-1ea0-4542-ab67-69e662133fee" (UID: "cf73b12d-1ea0-4542-ab67-69e662133fee"). InnerVolumeSpecName "kube-api-access-pdwbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:08.900223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:08.900197 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca" pod "cf73b12d-1ea0-4542-ab67-69e662133fee" (UID: "cf73b12d-1ea0-4542-ab67-69e662133fee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:09.003200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:09.003165 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdwbf\" (UniqueName: \"kubernetes.io/projected/cf73b12d-1ea0-4542-ab67-69e662133fee-kube-api-access-pdwbf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.003580 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:09.003560 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.003787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:09.003774 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf73b12d-1ea0-4542-ab67-69e662133fee-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:09.607099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:09.607069 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" Oct 30 05:50:09.607099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:09.607076 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m" event={"ID":"cf73b12d-1ea0-4542-ab67-69e662133fee","Type":"ContainerDied","Data":"80abc3c5b53fd59f7c6d7f139b86d6b8f895950502a474a531b4cbc63dff3bc6"} Oct 30 05:50:09.607285 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:09.607145 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80abc3c5b53fd59f7c6d7f139b86d6b8f895950502a474a531b4cbc63dff3bc6" Oct 30 05:50:12.137556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.134629 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl"] Oct 30 05:50:12.137556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.135732 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cf73b12d-1ea0-4542-ab67-69e662133fee" containerName="revoke-keys" Oct 30 05:50:12.137556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.135753 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf73b12d-1ea0-4542-ab67-69e662133fee" containerName="revoke-keys" Oct 30 05:50:12.137556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.135907 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf73b12d-1ea0-4542-ab67-69e662133fee" containerName="revoke-keys" Oct 30 05:50:12.433601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.432256 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl"] Oct 30 05:50:12.433601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.432405 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.499661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.490924 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:12.499661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.491478 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:12.536669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.536635 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:12.560553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.556499 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:50:12.560553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.557942 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.560553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.558046 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85mcz\" (UniqueName: \"kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.560553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.558161 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.692613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.690933 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85mcz\" (UniqueName: \"kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.692613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.691098 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.692613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.691288 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.692613 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.692181 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.757905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.757869 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85mcz\" (UniqueName: \"kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.806989 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.806956 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-23daeeca-jbbwl\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:12.889697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.889656 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:12.941662 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.941613 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:12.945600 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.942009 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:50:12.986960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:12.986917 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.076833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.076794 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:13.241611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.203234 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:50:13.398558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.396660 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:13.500808 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.500674 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.944680 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.945316 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.945621 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.945683 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.946281 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.946340 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:13.946375 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.026680 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.045605 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.054859 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.058880 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:50:14.106673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.060611 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.110568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.109160 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.407287 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.276817 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.407287 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.277988 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:50:14.479158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.444364 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": context deadline exceeded" Oct 30 05:50:14.479158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.444498 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded" Oct 30 05:50:14.479158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.445585 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.533031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.532981 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:14.774566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.535166 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:50:14.774566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:14.769321 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:50:15.001553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.000946 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:50:15.026616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.025648 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 05:50:15.026616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.025894 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:50:15.050355 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.027265 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded" Oct 30 05:50:15.087754 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.087704 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.088726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.088699 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:50:15.095164 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.091784 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:50:15.095164 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.093017 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:50:15.119559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.118092 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:50:15.123789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.123757 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:50:15.185552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.154311 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.222552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.197838 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.223409 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.223353 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.224032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.224013 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:50:15.224257 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.224241 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:50:15.315785 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.281212 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.477548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.477125 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.625633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.606753 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.811751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.811714 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:50:15.983995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.983915 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl"] Oct 30 05:50:15.987081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.987058 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:50:15.988501 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.988478 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" event={"ID":"8222e6f0-536f-43b1-aecf-238ca0a1e03b","Type":"ContainerStarted","Data":"db4a3bec677e30a625381f910b1fdf4d23460b0307da3b7cc71af074359d288b"} Oct 30 05:50:15.996936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:15.996914 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:50:16.001145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.001116 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:50:16.001310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.001278 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:50:16.011993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.011968 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:50:16.018079 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.018059 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:50:16.028812 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.028780 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded" Oct 30 05:50:16.073066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.073033 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:50:16.074598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.074574 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:50:16.077332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.077310 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:50:16.078034 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.078009 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:50:16.078159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.078062 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:50:16.078159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.078096 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:50:16.088184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.085634 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:50:16.998171 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:16.998139 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" event={"ID":"8222e6f0-536f-43b1-aecf-238ca0a1e03b","Type":"ContainerStarted","Data":"92486031bcc481e7701f78cb3ad987caa2c433b69f063cd7a4147a7d33c16c62"} Oct 30 05:50:17.044015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:17.043959 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" podStartSLOduration=5.043943362 podStartE2EDuration="5.043943362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:17.041256777 +0000 UTC m=+1123.616943833" watchObservedRunningTime="2026-10-30 05:50:17.043943362 +0000 UTC m=+1123.619630417" Oct 30 05:50:17.538123 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:50:17.538085 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:50:18.011458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:18.011425 5565 generic.go:356] "Generic (PLEG): container finished" podID="8222e6f0-536f-43b1-aecf-238ca0a1e03b" containerID="92486031bcc481e7701f78cb3ad987caa2c433b69f063cd7a4147a7d33c16c62" exitCode=0 Oct 30 05:50:18.011930 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:18.011481 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" event={"ID":"8222e6f0-536f-43b1-aecf-238ca0a1e03b","Type":"ContainerDied","Data":"92486031bcc481e7701f78cb3ad987caa2c433b69f063cd7a4147a7d33c16c62"} Oct 30 05:50:19.174231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.174208 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:19.253776 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.253745 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz\" (UniqueName: \"kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz\") pod \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " Oct 30 05:50:19.253962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.253942 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca\") pod \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " Oct 30 05:50:19.254151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.254129 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca\") pod \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\" (UID: \"8222e6f0-536f-43b1-aecf-238ca0a1e03b\") " Oct 30 05:50:19.254299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.254278 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca" pod "8222e6f0-536f-43b1-aecf-238ca0a1e03b" (UID: "8222e6f0-536f-43b1-aecf-238ca0a1e03b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:19.254860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.254834 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:19.256013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.255985 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz" pod "8222e6f0-536f-43b1-aecf-238ca0a1e03b" (UID: "8222e6f0-536f-43b1-aecf-238ca0a1e03b"). InnerVolumeSpecName "kube-api-access-85mcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:19.256144 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.256120 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca" pod "8222e6f0-536f-43b1-aecf-238ca0a1e03b" (UID: "8222e6f0-536f-43b1-aecf-238ca0a1e03b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:19.355690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.355631 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8222e6f0-536f-43b1-aecf-238ca0a1e03b-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:19.355690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:19.355655 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85mcz\" (UniqueName: \"kubernetes.io/projected/8222e6f0-536f-43b1-aecf-238ca0a1e03b-kube-api-access-85mcz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:20.030903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.030861 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" Oct 30 05:50:20.030903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.030879 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl" event={"ID":"8222e6f0-536f-43b1-aecf-238ca0a1e03b","Type":"ContainerDied","Data":"db4a3bec677e30a625381f910b1fdf4d23460b0307da3b7cc71af074359d288b"} Oct 30 05:50:20.031106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.030931 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="db4a3bec677e30a625381f910b1fdf4d23460b0307da3b7cc71af074359d288b" Oct 30 05:50:20.892442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.892409 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws"] Oct 30 05:50:20.893240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.893217 5565 cpu_manager.go:414] "containerMap: removing container" podUID="8222e6f0-536f-43b1-aecf-238ca0a1e03b" containerName="revoke-keys" Oct 30 05:50:20.893332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.893242 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8222e6f0-536f-43b1-aecf-238ca0a1e03b" containerName="revoke-keys" Oct 30 05:50:20.893473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.893459 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="8222e6f0-536f-43b1-aecf-238ca0a1e03b" containerName="revoke-keys" Oct 30 05:50:20.897488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.897462 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:20.914249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.914225 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:50:20.916874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.916851 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:20.927193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.927166 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:20.927319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.927217 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:20.933982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:20.933959 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws"] Oct 30 05:50:21.079850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.079812 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.080082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.079986 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.080082 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.080022 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjg52\" (UniqueName: \"kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.181684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.181585 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.181866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.181767 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.181866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.181814 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjg52\" (UniqueName: \"kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.182483 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.182459 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.184460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.184437 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.206834 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.206804 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjg52\" (UniqueName: \"kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52\") pod \"maas-api-revoke-sub-6b0c89d9-lw8ws\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.214185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.214152 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:21.535751 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:21.535715 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757ec86f_6b5b_479c_b7fb_c7ae9f44f2bf.slice/crio-f5672c5088810a29dcceebca3c5cd096f4d83edb56d6b4e5552c49fa4ef88e91 WatchSource:0}: Error finding container f5672c5088810a29dcceebca3c5cd096f4d83edb56d6b4e5552c49fa4ef88e91: Status 404 returned error can't find the container with id f5672c5088810a29dcceebca3c5cd096f4d83edb56d6b4e5552c49fa4ef88e91 Oct 30 05:50:21.536702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:21.536673 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws"] Oct 30 05:50:22.050959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:22.050924 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" event={"ID":"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf","Type":"ContainerStarted","Data":"7494312ccab1a22d11b97001cbffdfb88c25571fcacc550c3a5f36dd6b5fbbaa"} Oct 30 05:50:22.051378 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:22.051007 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" event={"ID":"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf","Type":"ContainerStarted","Data":"f5672c5088810a29dcceebca3c5cd096f4d83edb56d6b4e5552c49fa4ef88e91"} Oct 30 05:50:23.061135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:23.061092 5565 generic.go:356] "Generic (PLEG): container finished" podID="757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" containerID="7494312ccab1a22d11b97001cbffdfb88c25571fcacc550c3a5f36dd6b5fbbaa" exitCode=0 Oct 30 05:50:23.061611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:23.061163 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" event={"ID":"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf","Type":"ContainerDied","Data":"7494312ccab1a22d11b97001cbffdfb88c25571fcacc550c3a5f36dd6b5fbbaa"} Oct 30 05:50:24.260625 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.260602 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:24.335226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.335151 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52\" (UniqueName: \"kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52\") pod \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " Oct 30 05:50:24.335226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.335210 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca\") pod \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " Oct 30 05:50:24.335449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.335364 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca\") pod \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\" (UID: \"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf\") " Oct 30 05:50:24.335822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.335786 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca" pod "757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" (UID: "757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:24.336194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.336170 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:24.337344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.337316 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca" pod "757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" (UID: "757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:24.337804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.337777 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52" pod "757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" (UID: "757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf"). InnerVolumeSpecName "kube-api-access-tjg52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:24.381042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.381010 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9"] Oct 30 05:50:24.381708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.381689 5565 cpu_manager.go:414] "containerMap: removing container" podUID="757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" containerName="revoke-keys" Oct 30 05:50:24.381796 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.381708 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" containerName="revoke-keys" Oct 30 05:50:24.381931 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.381917 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" containerName="revoke-keys" Oct 30 05:50:24.385806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.385787 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.406748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.406677 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-6a881dd9\"" Oct 30 05:50:24.413217 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.413191 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9"] Oct 30 05:50:24.420400 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.420376 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-6a881dd9-dockercfg-gq7mk\"" Oct 30 05:50:24.439593 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.430821 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f"] Oct 30 05:50:24.440910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.440886 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2f6g\" (UniqueName: \"kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g\") pod \"payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.441204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.441174 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.441706 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.441685 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjg52\" (UniqueName: \"kubernetes.io/projected/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-kube-api-access-tjg52\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:24.441796 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.441710 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:24.445916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.445899 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.467084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.467059 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f"] Oct 30 05:50:24.543405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.543362 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.543599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.543557 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2f6g\" (UniqueName: \"kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g\") pod \"payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.543662 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.543629 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.543709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.543693 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvtdj\" (UniqueName: \"kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj\") pod \"payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.544761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.544736 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.583492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.583451 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2f6g\" (UniqueName: \"kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g\") pod \"payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.645444 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.645354 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.645624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.645537 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvtdj\" (UniqueName: \"kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj\") pod \"payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.646381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.646355 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume\") pod \"payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.672854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.672827 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvtdj\" (UniqueName: \"kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj\") pod \"payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:24.699467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.699435 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:24.773182 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:24.762630 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:25.017139 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:25.017109 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod373debca_9da7_4ca6_9d2b_0df853e14b9e.slice/crio-831b1b76177ee269b3bfa599482419aca2641441325aa240666c5bd851f0495e WatchSource:0}: Error finding container 831b1b76177ee269b3bfa599482419aca2641441325aa240666c5bd851f0495e: Status 404 returned error can't find the container with id 831b1b76177ee269b3bfa599482419aca2641441325aa240666c5bd851f0495e Oct 30 05:50:25.019776 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:25.019751 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9"] Oct 30 05:50:25.082976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:25.082944 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" Oct 30 05:50:25.083103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:25.082975 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws" event={"ID":"757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf","Type":"ContainerDied","Data":"f5672c5088810a29dcceebca3c5cd096f4d83edb56d6b4e5552c49fa4ef88e91"} Oct 30 05:50:25.083103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:25.083032 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f5672c5088810a29dcceebca3c5cd096f4d83edb56d6b4e5552c49fa4ef88e91" Oct 30 05:50:25.085172 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:25.085145 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" event={"ID":"373debca-9da7-4ca6-9d2b-0df853e14b9e","Type":"ContainerStarted","Data":"831b1b76177ee269b3bfa599482419aca2641441325aa240666c5bd851f0495e"} Oct 30 05:50:25.149826 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:25.149801 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fa5d4d0_76b8_4279_b578_54858e6c7399.slice/crio-fd05041a6c66b9bfaae93ff54f7e5644542025537fe695a7348434198176804b WatchSource:0}: Error finding container fd05041a6c66b9bfaae93ff54f7e5644542025537fe695a7348434198176804b: Status 404 returned error can't find the container with id fd05041a6c66b9bfaae93ff54f7e5644542025537fe695a7348434198176804b Oct 30 05:50:25.151293 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:25.151259 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f"] Oct 30 05:50:26.103547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.101806 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" event={"ID":"373debca-9da7-4ca6-9d2b-0df853e14b9e","Type":"ContainerStarted","Data":"9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d"} Oct 30 05:50:26.103547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.101891 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:26.103547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.103392 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" event={"ID":"0fa5d4d0-76b8-4279-b578-54858e6c7399","Type":"ContainerStarted","Data":"9959cded293ae073507f8deef381e1dc9ade5af7a2d83bbbc3d35754adff26e5"} Oct 30 05:50:26.103547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.103444 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" event={"ID":"0fa5d4d0-76b8-4279-b578-54858e6c7399","Type":"ContainerStarted","Data":"fd05041a6c66b9bfaae93ff54f7e5644542025537fe695a7348434198176804b"} Oct 30 05:50:26.103547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.103463 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:26.144888 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.144840 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" podStartSLOduration=2.144826121 podStartE2EDuration="2.144826121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:26.143026816 +0000 UTC m=+1132.718713872" watchObservedRunningTime="2026-10-30 05:50:26.144826121 +0000 UTC m=+1132.720513175" Oct 30 05:50:26.186850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:26.186801 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" podStartSLOduration=2.186788415 podStartE2EDuration="2.186788415s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:26.185595402 +0000 UTC m=+1132.761282458" watchObservedRunningTime="2026-10-30 05:50:26.186788415 +0000 UTC m=+1132.762475469" Oct 30 05:50:31.205050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.205017 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf"] Oct 30 05:50:31.210072 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.210050 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.225641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.225615 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:50:31.232798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.232774 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf"] Oct 30 05:50:31.238279 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.238246 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:50:31.252872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.252852 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:50:31.252872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.252858 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:50:31.344991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.344964 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.345120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.345098 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7dd\" (UniqueName: \"kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.345187 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.345129 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.446809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.446783 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7dd\" (UniqueName: \"kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.446965 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.446816 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.446965 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.446952 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.447581 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.447520 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.449097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.449078 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.471206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.471147 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7dd\" (UniqueName: \"kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd\") pod \"maas-api-revoke-sub-aa7ce029-t8zjf\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.525362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.525342 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:31.832810 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:31.832779 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4c094c9d_05cd_4697_bfed_b3b6c2a15ade.slice/crio-c21678614241f7221c0fc107a76c415a0b4ad2b32326e26150db0b637e8dbc91 WatchSource:0}: Error finding container c21678614241f7221c0fc107a76c415a0b4ad2b32326e26150db0b637e8dbc91: Status 404 returned error can't find the container with id c21678614241f7221c0fc107a76c415a0b4ad2b32326e26150db0b637e8dbc91 Oct 30 05:50:31.833792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:31.833769 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf"] Oct 30 05:50:32.159021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:32.158988 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" event={"ID":"4c094c9d-05cd-4697-bfed-b3b6c2a15ade","Type":"ContainerStarted","Data":"c21678614241f7221c0fc107a76c415a0b4ad2b32326e26150db0b637e8dbc91"} Oct 30 05:50:32.917264 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:32.917237 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7"] Oct 30 05:50:32.921861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:32.921835 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:32.953482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:32.953452 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7"] Oct 30 05:50:33.069638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069522 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.069763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069639 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.069763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069720 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.069763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069749 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnqm7\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.069906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069784 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.069948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069909 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.069994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.069976 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.070096 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.070080 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.070162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.070148 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.170321 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.170287 5565 generic.go:356] "Generic (PLEG): container finished" podID="4c094c9d-05cd-4697-bfed-b3b6c2a15ade" containerID="05acadc1cbcbd33fd3bfe2eed62b8d57030c45e3e2a117366e6fb859869a66fb" exitCode=0 Oct 30 05:50:33.170492 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.170330 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" event={"ID":"4c094c9d-05cd-4697-bfed-b3b6c2a15ade","Type":"ContainerDied","Data":"05acadc1cbcbd33fd3bfe2eed62b8d57030c45e3e2a117366e6fb859869a66fb"} Oct 30 05:50:33.171021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.170998 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171131 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171099 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171195 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171166 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171279 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171266 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171357 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171341 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171358 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171406 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171467 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171514 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnqm7\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171578 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171635 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171716 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.171774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.171762 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.172229 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.172209 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.173751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.173733 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.174058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.174037 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.194823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.194795 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.195556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.195517 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnqm7\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7\") pod \"e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.243393 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.243370 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:33.571357 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:33.571310 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf538d252_0856_4886_823a_1cd114d0f319.slice/crio-b93c85df77874f7e590aa6cc93d47b3361881d0babe9e30c2391f82b74b786e3 WatchSource:0}: Error finding container b93c85df77874f7e590aa6cc93d47b3361881d0babe9e30c2391f82b74b786e3: Status 404 returned error can't find the container with id b93c85df77874f7e590aa6cc93d47b3361881d0babe9e30c2391f82b74b786e3 Oct 30 05:50:33.572733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.572707 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7"] Oct 30 05:50:33.573770 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.573740 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:50:33.573864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.573798 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:50:33.573864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:33.573827 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:50:34.190240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.190197 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" event={"ID":"f538d252-0856-4886-823a-1cd114d0f319","Type":"ContainerStarted","Data":"2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975"} Oct 30 05:50:34.190729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.190293 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" event={"ID":"f538d252-0856-4886-823a-1cd114d0f319","Type":"ContainerStarted","Data":"b93c85df77874f7e590aa6cc93d47b3361881d0babe9e30c2391f82b74b786e3"} Oct 30 05:50:34.240557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.240474 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podStartSLOduration=2.240460163 podStartE2EDuration="2.240460163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:34.238935634 +0000 UTC m=+1140.814622707" watchObservedRunningTime="2026-10-30 05:50:34.240460163 +0000 UTC m=+1140.816147288" Oct 30 05:50:34.243887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.243857 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:34.245706 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.245682 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:34.245841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.245731 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:34.387899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.387871 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:34.499290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.499200 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd\" (UniqueName: \"kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd\") pod \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " Oct 30 05:50:34.499290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.499284 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca\") pod \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " Oct 30 05:50:34.499540 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.499415 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca\") pod \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\" (UID: \"4c094c9d-05cd-4697-bfed-b3b6c2a15ade\") " Oct 30 05:50:34.499767 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.499736 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca" pod "4c094c9d-05cd-4697-bfed-b3b6c2a15ade" (UID: "4c094c9d-05cd-4697-bfed-b3b6c2a15ade"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:50:34.500186 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.500159 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:34.501647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.501622 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd" pod "4c094c9d-05cd-4697-bfed-b3b6c2a15ade" (UID: "4c094c9d-05cd-4697-bfed-b3b6c2a15ade"). InnerVolumeSpecName "kube-api-access-lf7dd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:50:34.501746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.501710 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca" pod "4c094c9d-05cd-4697-bfed-b3b6c2a15ade" (UID: "4c094c9d-05cd-4697-bfed-b3b6c2a15ade"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:50:34.601895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.601864 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:34.601895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:34.601894 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7dd\" (UniqueName: \"kubernetes.io/projected/4c094c9d-05cd-4697-bfed-b3b6c2a15ade-kube-api-access-lf7dd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:50:35.202036 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:35.202005 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" Oct 30 05:50:35.202414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:35.202041 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf" event={"ID":"4c094c9d-05cd-4697-bfed-b3b6c2a15ade","Type":"ContainerDied","Data":"c21678614241f7221c0fc107a76c415a0b4ad2b32326e26150db0b637e8dbc91"} Oct 30 05:50:35.202414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:35.202102 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c21678614241f7221c0fc107a76c415a0b4ad2b32326e26150db0b637e8dbc91" Oct 30 05:50:35.244106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:35.244079 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:35.244214 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:35.244133 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:36.244101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:36.244070 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:36.244523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:36.244145 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:37.116051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:37.116024 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:50:37.243684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:37.243651 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:37.243840 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:37.243700 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:38.123413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:38.123383 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:50:38.243985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:38.243950 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:38.244158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:38.244013 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:39.243633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:39.243605 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:39.244020 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:39.243652 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:40.244071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:40.244034 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" start-of-body= Oct 30 05:50:40.244444 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:40.244089 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.119:15021/healthz/ready\": dial tcp 10.134.0.119:15021: connect: connection refused" Oct 30 05:50:41.248446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:41.248415 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:41.248904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:41.248827 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:41.249430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:41.249413 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 05:50:44.108408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:44.108365 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:50:47.695949 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:50:47.695907 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:50:49.176071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.176037 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5"] Oct 30 05:50:49.176728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.176712 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4c094c9d-05cd-4697-bfed-b3b6c2a15ade" containerName="revoke-keys" Oct 30 05:50:49.176728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.176729 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4c094c9d-05cd-4697-bfed-b3b6c2a15ade" containerName="revoke-keys" Oct 30 05:50:49.176838 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.176828 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4c094c9d-05cd-4697-bfed-b3b6c2a15ade" containerName="revoke-keys" Oct 30 05:50:49.181892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.181875 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.201882 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.201860 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5"] Oct 30 05:50:49.208791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.208760 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-19e33488\"" Oct 30 05:50:49.208945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.208925 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-19e33488-dockercfg-twjm9\"" Oct 30 05:50:49.248100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.248070 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976"] Oct 30 05:50:49.252873 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.252853 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.273154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.273126 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976"] Oct 30 05:50:49.309487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.309440 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gc72k\" (UniqueName: \"kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k\") pod \"payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.309694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.309511 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.309781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.309730 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-945nd\" (UniqueName: \"kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd\") pod \"payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.309864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.309842 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.410788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.410755 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-945nd\" (UniqueName: \"kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd\") pod \"payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.410973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.410851 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.410973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.410929 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gc72k\" (UniqueName: \"kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k\") pod \"payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.410973 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.410951 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.411688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.411662 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume\") pod \"payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.411793 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.411718 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.435442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.435374 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-945nd\" (UniqueName: \"kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd\") pod \"payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.435614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.435477 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gc72k\" (UniqueName: \"kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k\") pod \"payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.495672 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.495639 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:49.567300 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.567099 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:49.814474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.814446 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5"] Oct 30 05:50:49.920583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:49.920550 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976"] Oct 30 05:50:49.924859 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:50:49.924831 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3102109e_9eb0_479f_8b72_12291b0305ed.slice/crio-0958c082b6316236f20ec56b0bc4aecf37903ba0e40533603856a15950559edc WatchSource:0}: Error finding container 0958c082b6316236f20ec56b0bc4aecf37903ba0e40533603856a15950559edc: Status 404 returned error can't find the container with id 0958c082b6316236f20ec56b0bc4aecf37903ba0e40533603856a15950559edc Oct 30 05:50:50.401748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.401704 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" event={"ID":"3102109e-9eb0-479f-8b72-12291b0305ed","Type":"ContainerStarted","Data":"41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0"} Oct 30 05:50:50.402181 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.401767 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" event={"ID":"3102109e-9eb0-479f-8b72-12291b0305ed","Type":"ContainerStarted","Data":"0958c082b6316236f20ec56b0bc4aecf37903ba0e40533603856a15950559edc"} Oct 30 05:50:50.402181 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.401800 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:50:50.403560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.403504 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" event={"ID":"8e9f6598-fb41-4f5a-833f-d705aec48459","Type":"ContainerStarted","Data":"64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b"} Oct 30 05:50:50.403673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.403584 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:50:50.403673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.403610 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" event={"ID":"8e9f6598-fb41-4f5a-833f-d705aec48459","Type":"ContainerStarted","Data":"e92ec379b7ca884ba36b1d4c6b88b94bcdc2b9c58286c511aab432e4d97a51d5"} Oct 30 05:50:50.451756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.451706 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" podStartSLOduration=1.4516926780000001 podStartE2EDuration="1.451692678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:50.445104617 +0000 UTC m=+1157.020791674" watchObservedRunningTime="2026-10-30 05:50:50.451692678 +0000 UTC m=+1157.027379765" Oct 30 05:50:50.487485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:50.487438 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" podStartSLOduration=1.487427691 podStartE2EDuration="1.487427691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:50:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:50:50.48632126 +0000 UTC m=+1157.062008314" watchObservedRunningTime="2026-10-30 05:50:50.487427691 +0000 UTC m=+1157.063114745" Oct 30 05:50:58.043044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:58.043011 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh"] Oct 30 05:50:58.059502 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:58.059474 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e870778b-6s6vh"] Oct 30 05:50:59.568466 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:50:59.568435 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e849a6b2-593e-43b6-a52a-f4e58b6b2126" path="/var/lib/kubelet/pods/e849a6b2-593e-43b6-a52a-f4e58b6b2126/volumes" Oct 30 05:51:01.415309 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:01.415279 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:51:01.415853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:01.415352 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:51:03.974798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:03.974746 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n"] Oct 30 05:51:03.980947 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:03.980919 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.001957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.001918 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n"] Oct 30 05:51:04.007038 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.007007 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:04.007302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.007068 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:04.009713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.009685 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:04.010033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.010004 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:51:04.020824 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.020778 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.021024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.020851 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.021077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.021038 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbwqq\" (UniqueName: \"kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.122927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.122878 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbwqq\" (UniqueName: \"kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.123155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.123041 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.123155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.123080 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.123804 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.123777 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.125699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.125680 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.147493 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.147461 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbwqq\" (UniqueName: \"kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq\") pod \"maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.298074 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.298026 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:04.615621 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:04.615581 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod869641eb_b20d_4c1b_a3a1_bf2c33cbfeb5.slice/crio-ba1937e4dbec91660f662852332734015e504373ce7576294b3fc03e2e5c84e1 WatchSource:0}: Error finding container ba1937e4dbec91660f662852332734015e504373ce7576294b3fc03e2e5c84e1: Status 404 returned error can't find the container with id ba1937e4dbec91660f662852332734015e504373ce7576294b3fc03e2e5c84e1 Oct 30 05:51:04.618198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:04.618167 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n"] Oct 30 05:51:05.542981 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:05.542926 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" event={"ID":"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5","Type":"ContainerStarted","Data":"6928a3a025e7d30787093222274bc60bb9f8b648dcf74689db6dd34a35e74a90"} Oct 30 05:51:05.543797 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:05.543037 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" event={"ID":"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5","Type":"ContainerStarted","Data":"ba1937e4dbec91660f662852332734015e504373ce7576294b3fc03e2e5c84e1"} Oct 30 05:51:05.591298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:05.591228 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" podStartSLOduration=2.591210905 podStartE2EDuration="2.591210905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:05.585738014 +0000 UTC m=+1172.161425082" watchObservedRunningTime="2026-10-30 05:51:05.591210905 +0000 UTC m=+1172.166897958" Oct 30 05:51:06.553982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:06.553946 5565 generic.go:356] "Generic (PLEG): container finished" podID="869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" containerID="6928a3a025e7d30787093222274bc60bb9f8b648dcf74689db6dd34a35e74a90" exitCode=0 Oct 30 05:51:06.554396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:06.554024 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" event={"ID":"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5","Type":"ContainerDied","Data":"6928a3a025e7d30787093222274bc60bb9f8b648dcf74689db6dd34a35e74a90"} Oct 30 05:51:07.037210 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.037173 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s"] Oct 30 05:51:07.044762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.044727 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-caef318f-z9k8s"] Oct 30 05:51:07.568763 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.568728 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa101b13-4b16-45ad-b411-44d2db791dc3" path="/var/lib/kubelet/pods/aa101b13-4b16-45ad-b411-44d2db791dc3/volumes" Oct 30 05:51:07.748603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.748575 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:07.884105 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.884009 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca\") pod \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " Oct 30 05:51:07.884308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.884134 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca\") pod \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " Oct 30 05:51:07.884308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.884219 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq\" (UniqueName: \"kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq\") pod \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\" (UID: \"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5\") " Oct 30 05:51:07.887259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.884793 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca" pod "869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" (UID: "869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:07.887484 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.887459 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca" pod "869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" (UID: "869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:07.888043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.888014 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:07.888139 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.888051 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:07.897342 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.897302 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq" pod "869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" (UID: "869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5"). InnerVolumeSpecName "kube-api-access-lbwqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:07.990107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:07.990072 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbwqq\" (UniqueName: \"kubernetes.io/projected/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5-kube-api-access-lbwqq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:08.572744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:08.572709 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" Oct 30 05:51:08.573153 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:08.572748 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n" event={"ID":"869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5","Type":"ContainerDied","Data":"ba1937e4dbec91660f662852332734015e504373ce7576294b3fc03e2e5c84e1"} Oct 30 05:51:08.573153 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:08.572799 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba1937e4dbec91660f662852332734015e504373ce7576294b3fc03e2e5c84e1" Oct 30 05:51:14.610620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:14.610569 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f"] Oct 30 05:51:14.645705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:14.645671 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9"] Oct 30 05:51:15.038653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:15.036415 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8"] Oct 30 05:51:15.056471 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:15.056420 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb"] Oct 30 05:51:15.061920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:15.061878 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aea364b4-vqwd8"] Oct 30 05:51:15.067666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:15.067627 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0275c3e1-2kmtb"] Oct 30 05:51:15.571839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:15.571795 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6507588a-cdd2-4f2a-9133-b34def476866" path="/var/lib/kubelet/pods/6507588a-cdd2-4f2a-9133-b34def476866/volumes" Oct 30 05:51:15.574053 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:15.574016 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f331dc96-299e-4ef6-8a81-ed75cca5a097" path="/var/lib/kubelet/pods/f331dc96-299e-4ef6-8a81-ed75cca5a097/volumes" Oct 30 05:51:17.541161 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:51:17.541124 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:51:17.991154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:17.991065 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q"] Oct 30 05:51:17.992071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:17.992053 5565 cpu_manager.go:414] "containerMap: removing container" podUID="869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" containerName="revoke-keys" Oct 30 05:51:17.992131 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:17.992074 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" containerName="revoke-keys" Oct 30 05:51:17.992281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:17.992261 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" containerName="revoke-keys" Oct 30 05:51:17.998694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:17.998657 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.002821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.002740 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q"] Oct 30 05:51:18.016880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.016851 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:18.017070 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.016961 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:51:18.017070 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.017067 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:18.017255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.017238 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:18.111055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.111026 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmvr\" (UniqueName: \"kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.111265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.111095 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.111265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.111213 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.216219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.215462 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znmvr\" (UniqueName: \"kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.216219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.215622 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.216219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.215686 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.216556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.216384 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.218856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.218829 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.239380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.239356 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmvr\" (UniqueName: \"kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr\") pod \"maas-api-revoke-sub-e8ed2e49-hcw4q\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.256187 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.256158 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg"] Oct 30 05:51:18.263148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.262185 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.284043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.284017 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg"] Oct 30 05:51:18.295582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.295219 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2"] Oct 30 05:51:18.301043 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.301018 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322366 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg8wr\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322488 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322562 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322705 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322848 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322893 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322943 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.322993 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.323038 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.325647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.323392 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:18.360615 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.360580 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2"] Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425038 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425124 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425178 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425216 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425282 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425332 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg8wr\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425411 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425488 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425565 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425598 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425652 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.425757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425741 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425828 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425881 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.425926 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvjjs\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.426027 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.426074 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.426124 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.426567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.426168 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.426935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.426600 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.426935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.426679 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.427073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.427048 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.427311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.427286 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.429780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.429759 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.431777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.431730 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.451188 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.451121 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg8wr\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.452021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.451993 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.527817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.527731 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.527960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.527825 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.527960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.527867 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvjjs\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.527960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.527951 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528026 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528072 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528096 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528144 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528186 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528252 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528457 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528429 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528564 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.528673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.528645 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.529073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.529046 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.530683 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.530655 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.530800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.530694 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.551864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.551838 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.552259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.552123 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvjjs\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs\") pod \"e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.578522 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.578499 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:18.660611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.657834 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:18.709587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.707896 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q"] Oct 30 05:51:18.709587 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:18.709162 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26c0ceba_2a05_4678_9d17_82dee819b5bb.slice/crio-5aba2cec1beae10d6f91ea63e2542bd3ab9cc77dbf6974471652171ba95a623d WatchSource:0}: Error finding container 5aba2cec1beae10d6f91ea63e2542bd3ab9cc77dbf6974471652171ba95a623d: Status 404 returned error can't find the container with id 5aba2cec1beae10d6f91ea63e2542bd3ab9cc77dbf6974471652171ba95a623d Oct 30 05:51:18.764484 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.749473 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" event={"ID":"26c0ceba-2a05-4678-9d17-82dee819b5bb","Type":"ContainerStarted","Data":"5aba2cec1beae10d6f91ea63e2542bd3ab9cc77dbf6974471652171ba95a623d"} Oct 30 05:51:18.910388 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.910351 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:51:18.910574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.910442 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:51:18.910574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.910487 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:51:18.918416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:18.918379 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg"] Oct 30 05:51:19.023863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.022456 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2"] Oct 30 05:51:19.030816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.024373 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:51:19.030816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.024454 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:51:19.030816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.024495 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:51:19.597710 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.597677 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n"] Oct 30 05:51:19.602925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.602898 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-6a881dd9-vc29n"] Oct 30 05:51:19.612004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.611975 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" podUID="0fa5d4d0-76b8-4279-b578-54858e6c7399" containerName="payload-processing" containerID="cri-o://9959cded293ae073507f8deef381e1dc9ade5af7a2d83bbbc3d35754adff26e5" gracePeriod=25 Oct 30 05:51:19.713549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.704665 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" podUID="373debca-9da7-4ca6-9d2b-0df853e14b9e" containerName="payload-pre-processing" containerID="cri-o://9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d" gracePeriod=25 Oct 30 05:51:19.842096 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.842058 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" event={"ID":"26c0ceba-2a05-4678-9d17-82dee819b5bb","Type":"ContainerStarted","Data":"d9af6b6172e067b75464feb1980f54969566bcee09480cfe8b9c006c35c4de51"} Oct 30 05:51:19.844735 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.844707 5565 generic.go:356] "Generic (PLEG): container finished" podID="0fa5d4d0-76b8-4279-b578-54858e6c7399" containerID="9959cded293ae073507f8deef381e1dc9ade5af7a2d83bbbc3d35754adff26e5" exitCode=0 Oct 30 05:51:19.844866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.844776 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" event={"ID":"0fa5d4d0-76b8-4279-b578-54858e6c7399","Type":"ContainerDied","Data":"9959cded293ae073507f8deef381e1dc9ade5af7a2d83bbbc3d35754adff26e5"} Oct 30 05:51:19.846846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.846761 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" event={"ID":"de36eef2-9a73-4600-848e-650ff0c1612f","Type":"ContainerStarted","Data":"649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37"} Oct 30 05:51:19.846846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.846813 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" event={"ID":"de36eef2-9a73-4600-848e-650ff0c1612f","Type":"ContainerStarted","Data":"54f29f1c7cb4c0a127c30ebcb65f8b114a28950ce9b66c4af7fd8cd8b7c71433"} Oct 30 05:51:19.880606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.861802 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" event={"ID":"99a0f929-f527-4920-ac51-6c0fbbfea9a9","Type":"ContainerStarted","Data":"b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100"} Oct 30 05:51:19.880606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:19.861859 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" event={"ID":"99a0f929-f527-4920-ac51-6c0fbbfea9a9","Type":"ContainerStarted","Data":"11edf119783cbc9dfa0dd755e1cfa11e1a9ef0d9d60cdda351fe4aaf3f175ce0"} Oct 30 05:51:20.138673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.131147 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podStartSLOduration=2.131123001 podStartE2EDuration="2.131123001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:20.053809779 +0000 UTC m=+1186.629496834" watchObservedRunningTime="2026-10-30 05:51:20.131123001 +0000 UTC m=+1186.706810056" Oct 30 05:51:20.448380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.448286 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podStartSLOduration=2.448255078 podStartE2EDuration="2.448255078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:20.003141305 +0000 UTC m=+1186.578828359" watchObservedRunningTime="2026-10-30 05:51:20.448255078 +0000 UTC m=+1187.023942133" Oct 30 05:51:20.534585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.533410 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:51:20.550758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.550426 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj\" (UniqueName: \"kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj\") pod \"0fa5d4d0-76b8-4279-b578-54858e6c7399\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " Oct 30 05:51:20.550758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.550661 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume\") pod \"0fa5d4d0-76b8-4279-b578-54858e6c7399\" (UID: \"0fa5d4d0-76b8-4279-b578-54858e6c7399\") " Oct 30 05:51:20.551421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.551377 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume" pod "0fa5d4d0-76b8-4279-b578-54858e6c7399" (UID: "0fa5d4d0-76b8-4279-b578-54858e6c7399"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:20.629253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.628472 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:20.629732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.629666 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:51:20.629732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.629717 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 05:51:20.630156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.630125 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj" pod "0fa5d4d0-76b8-4279-b578-54858e6c7399" (UID: "0fa5d4d0-76b8-4279-b578-54858e6c7399"). InnerVolumeSpecName "kube-api-access-rvtdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:20.660553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.653017 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fa5d4d0-76b8-4279-b578-54858e6c7399-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:20.660553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.653049 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvtdj\" (UniqueName: \"kubernetes.io/projected/0fa5d4d0-76b8-4279-b578-54858e6c7399-kube-api-access-rvtdj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:20.704582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.704521 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:20.709180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.709149 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 05:51:20.709413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.709383 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 05:51:20.828652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.828624 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:51:20.864539 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.864105 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g\" (UniqueName: \"kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g\") pod \"373debca-9da7-4ca6-9d2b-0df853e14b9e\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " Oct 30 05:51:20.864539 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.864345 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume\") pod \"373debca-9da7-4ca6-9d2b-0df853e14b9e\" (UID: \"373debca-9da7-4ca6-9d2b-0df853e14b9e\") " Oct 30 05:51:20.880587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.878775 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume" pod "373debca-9da7-4ca6-9d2b-0df853e14b9e" (UID: "373debca-9da7-4ca6-9d2b-0df853e14b9e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:21.001126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.985963 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/373debca-9da7-4ca6-9d2b-0df853e14b9e-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:21.001126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:20.986138 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g" pod "373debca-9da7-4ca6-9d2b-0df853e14b9e" (UID: "373debca-9da7-4ca6-9d2b-0df853e14b9e"). InnerVolumeSpecName "kube-api-access-t2f6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:21.127705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.127582 5565 generic.go:356] "Generic (PLEG): container finished" podID="373debca-9da7-4ca6-9d2b-0df853e14b9e" containerID="9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d" exitCode=0 Oct 30 05:51:21.128079 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.128006 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" event={"ID":"373debca-9da7-4ca6-9d2b-0df853e14b9e","Type":"ContainerDied","Data":"9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d"} Oct 30 05:51:21.128263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.128200 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" event={"ID":"373debca-9da7-4ca6-9d2b-0df853e14b9e","Type":"ContainerDied","Data":"831b1b76177ee269b3bfa599482419aca2641441325aa240666c5bd851f0495e"} Oct 30 05:51:21.128457 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.128385 5565 scope.go:122] "RemoveContainer" containerID="9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d" Oct 30 05:51:21.128813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.128754 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9" Oct 30 05:51:21.138405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.138379 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2f6g\" (UniqueName: \"kubernetes.io/projected/373debca-9da7-4ca6-9d2b-0df853e14b9e-kube-api-access-t2f6g\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:21.244342 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.242665 5565 generic.go:356] "Generic (PLEG): container finished" podID="26c0ceba-2a05-4678-9d17-82dee819b5bb" containerID="d9af6b6172e067b75464feb1980f54969566bcee09480cfe8b9c006c35c4de51" exitCode=0 Oct 30 05:51:21.244342 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.242814 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" event={"ID":"26c0ceba-2a05-4678-9d17-82dee819b5bb","Type":"ContainerDied","Data":"d9af6b6172e067b75464feb1980f54969566bcee09480cfe8b9c006c35c4de51"} Oct 30 05:51:21.248594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.246566 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" Oct 30 05:51:21.248594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.247469 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f" event={"ID":"0fa5d4d0-76b8-4279-b578-54858e6c7399","Type":"ContainerDied","Data":"fd05041a6c66b9bfaae93ff54f7e5644542025537fe695a7348434198176804b"} Oct 30 05:51:21.339637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.334978 5565 scope.go:122] "RemoveContainer" containerID="9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d" Oct 30 05:51:21.339637 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:51:21.335445 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d\": container with ID starting with 9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d not found: ID does not exist" containerID="9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d" Oct 30 05:51:21.339637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.335485 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d"} err="failed to get container status \"9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d\": rpc error: code = NotFound desc = could not find container \"9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d\": container with ID starting with 9675c27d5fea2290e34a7f05d057229c6f5d0ff3e6df74fdb78822680724e24d not found: ID does not exist" Oct 30 05:51:21.339637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.335510 5565 scope.go:122] "RemoveContainer" containerID="9959cded293ae073507f8deef381e1dc9ade5af7a2d83bbbc3d35754adff26e5" Oct 30 05:51:21.350779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.350517 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9"] Oct 30 05:51:21.430103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.430070 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-6a881dd9-f8d6b6f9-6k2h9"] Oct 30 05:51:21.437820 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.437793 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f"] Oct 30 05:51:21.442493 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.442441 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-6a881dd9-75555fb4f6-vd28f"] Oct 30 05:51:21.584607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.578972 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:51:21.637066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.637016 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 05:51:21.646049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.644395 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fa5d4d0-76b8-4279-b578-54858e6c7399" path="/var/lib/kubelet/pods/0fa5d4d0-76b8-4279-b578-54858e6c7399/volumes" Oct 30 05:51:21.646049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.645171 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="373debca-9da7-4ca6-9d2b-0df853e14b9e" path="/var/lib/kubelet/pods/373debca-9da7-4ca6-9d2b-0df853e14b9e/volumes" Oct 30 05:51:21.646049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.645800 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5" path="/var/lib/kubelet/pods/869641eb-b20d-4c1b-a3a1-bf2c33cbfeb5/volumes" Oct 30 05:51:21.674551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.665222 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 05:51:21.674551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.665283 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 05:51:21.808855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.808833 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:21.849560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.849463 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca\") pod \"26c0ceba-2a05-4678-9d17-82dee819b5bb\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " Oct 30 05:51:21.849738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.849583 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca\") pod \"26c0ceba-2a05-4678-9d17-82dee819b5bb\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " Oct 30 05:51:21.849738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.849629 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr\" (UniqueName: \"kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr\") pod \"26c0ceba-2a05-4678-9d17-82dee819b5bb\" (UID: \"26c0ceba-2a05-4678-9d17-82dee819b5bb\") " Oct 30 05:51:21.849935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.849909 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca" pod "26c0ceba-2a05-4678-9d17-82dee819b5bb" (UID: "26c0ceba-2a05-4678-9d17-82dee819b5bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:21.850504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.850481 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:21.882802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.874720 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr" pod "26c0ceba-2a05-4678-9d17-82dee819b5bb" (UID: "26c0ceba-2a05-4678-9d17-82dee819b5bb"). InnerVolumeSpecName "kube-api-access-znmvr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:21.882802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.874815 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca" pod "26c0ceba-2a05-4678-9d17-82dee819b5bb" (UID: "26c0ceba-2a05-4678-9d17-82dee819b5bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:21.952551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.952496 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znmvr\" (UniqueName: \"kubernetes.io/projected/26c0ceba-2a05-4678-9d17-82dee819b5bb-kube-api-access-znmvr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:21.952551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:21.952552 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26c0ceba-2a05-4678-9d17-82dee819b5bb-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:22.310867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.310839 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" Oct 30 05:51:22.311122 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.311093 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q" event={"ID":"26c0ceba-2a05-4678-9d17-82dee819b5bb","Type":"ContainerDied","Data":"5aba2cec1beae10d6f91ea63e2542bd3ab9cc77dbf6974471652171ba95a623d"} Oct 30 05:51:22.311241 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.311229 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5aba2cec1beae10d6f91ea63e2542bd3ab9cc77dbf6974471652171ba95a623d" Oct 30 05:51:22.592612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.592264 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:51:22.592612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.592336 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 05:51:22.698547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.697974 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 05:51:22.698547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:22.698041 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 05:51:23.584551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:23.583765 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:51:23.584551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:23.583824 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 05:51:23.671553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:23.660857 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 05:51:23.671553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:23.660918 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 05:51:24.594547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:24.594292 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:51:24.594547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:24.594366 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 05:51:24.659870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:24.659836 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 05:51:24.660114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:24.660087 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 05:51:25.579424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:25.579394 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" start-of-body= Oct 30 05:51:25.579654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:25.579455 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.124:15021/healthz/ready\": dial tcp 10.134.0.124:15021: connect: connection refused" Oct 30 05:51:25.658742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:25.658710 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 30 05:51:25.659254 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:25.658759 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 30 05:51:26.037410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.037377 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh"] Oct 30 05:51:26.050241 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.050211 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k"] Oct 30 05:51:26.055094 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.055067 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3552c301-9t4wh"] Oct 30 05:51:26.059104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.059078 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cbf9c38e-sbv5k"] Oct 30 05:51:26.406203 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.406109 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx"] Oct 30 05:51:26.407240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407212 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0fa5d4d0-76b8-4279-b578-54858e6c7399" containerName="payload-processing" Oct 30 05:51:26.407240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407239 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fa5d4d0-76b8-4279-b578-54858e6c7399" containerName="payload-processing" Oct 30 05:51:26.407445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407261 5565 cpu_manager.go:414] "containerMap: removing container" podUID="373debca-9da7-4ca6-9d2b-0df853e14b9e" containerName="payload-pre-processing" Oct 30 05:51:26.407445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407270 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="373debca-9da7-4ca6-9d2b-0df853e14b9e" containerName="payload-pre-processing" Oct 30 05:51:26.407445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407321 5565 cpu_manager.go:414] "containerMap: removing container" podUID="26c0ceba-2a05-4678-9d17-82dee819b5bb" containerName="revoke-keys" Oct 30 05:51:26.407445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407330 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26c0ceba-2a05-4678-9d17-82dee819b5bb" containerName="revoke-keys" Oct 30 05:51:26.407697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407522 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fa5d4d0-76b8-4279-b578-54858e6c7399" containerName="payload-processing" Oct 30 05:51:26.407697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407559 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="373debca-9da7-4ca6-9d2b-0df853e14b9e" containerName="payload-pre-processing" Oct 30 05:51:26.407697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.407590 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="26c0ceba-2a05-4678-9d17-82dee819b5bb" containerName="revoke-keys" Oct 30 05:51:26.412344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.412320 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.432048 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.430700 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:26.432048 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.430776 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:26.469560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.469478 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx"] Oct 30 05:51:26.475350 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.475307 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:26.475490 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.475423 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:51:26.535191 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.535156 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.535381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.535216 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc76s\" (UniqueName: \"kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.535653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.535626 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.638126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.638094 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.638558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.638509 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.638743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.638722 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc76s\" (UniqueName: \"kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.639474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.639441 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.641273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.641243 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.663552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.663486 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc76s\" (UniqueName: \"kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s\") pod \"maas-api-revoke-sub-dc38413d-ws9bx\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.740373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.739126 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:26.740373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.740197 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:26.740984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.740960 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 05:51:26.767103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.767065 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:26.860802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.858803 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:26.860802 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.860411 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:26.861423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:26.861193 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 05:51:27.089209 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:27.089175 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7366be08_ab4b_4d73_831a_7a0f081d582e.slice/crio-d558ff46ccbbc627b2ef30986e3256c1f35191c751f0d5fb4c8094e575c0bd7f WatchSource:0}: Error finding container d558ff46ccbbc627b2ef30986e3256c1f35191c751f0d5fb4c8094e575c0bd7f: Status 404 returned error can't find the container with id d558ff46ccbbc627b2ef30986e3256c1f35191c751f0d5fb4c8094e575c0bd7f Oct 30 05:51:27.093209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:27.093180 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx"] Oct 30 05:51:27.371777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:27.371744 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" event={"ID":"7366be08-ab4b-4d73-831a-7a0f081d582e","Type":"ContainerStarted","Data":"d558ff46ccbbc627b2ef30986e3256c1f35191c751f0d5fb4c8094e575c0bd7f"} Oct 30 05:51:27.424353 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:27.424308 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" podStartSLOduration=1.424295386 podStartE2EDuration="1.424295386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:27.420887533 +0000 UTC m=+1193.996574592" watchObservedRunningTime="2026-10-30 05:51:27.424295386 +0000 UTC m=+1193.999982439" Oct 30 05:51:27.568828 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:27.568796 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b24a52da-4a84-45aa-bd96-e3c6f53758d2" path="/var/lib/kubelet/pods/b24a52da-4a84-45aa-bd96-e3c6f53758d2/volumes" Oct 30 05:51:27.569298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:27.569278 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4995005-531f-4f80-91f5-e1d240dc4404" path="/var/lib/kubelet/pods/d4995005-531f-4f80-91f5-e1d240dc4404/volumes" Oct 30 05:51:28.382104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.382068 5565 generic.go:356] "Generic (PLEG): container finished" podID="7366be08-ab4b-4d73-831a-7a0f081d582e" containerID="9b48576af5a5d2adfab624213c165c94e2d9034ee5eae5d355ae3dda5c11ff29" exitCode=0 Oct 30 05:51:28.382561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.382156 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" event={"ID":"7366be08-ab4b-4d73-831a-7a0f081d582e","Type":"ContainerDied","Data":"9b48576af5a5d2adfab624213c165c94e2d9034ee5eae5d355ae3dda5c11ff29"} Oct 30 05:51:28.561201 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.561161 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f"] Oct 30 05:51:28.565709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.565691 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.588051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.588020 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f"] Oct 30 05:51:28.670736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.670655 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.670907 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.670832 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gp6q\" (UniqueName: \"kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.670962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.670943 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.772149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.772110 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.772337 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.772318 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.772491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.772475 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gp6q\" (UniqueName: \"kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.772876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.772857 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.774642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.774620 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.796338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.796315 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gp6q\" (UniqueName: \"kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q\") pod \"maas-api-revoke-sub-dc8f378f-f9d7f\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:28.880009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:28.879973 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:29.192943 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:29.192908 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc03814b_3458_4fb9_9b32_112310d3649e.slice/crio-eaac51585ffac5c88f8e71cb5b5c4982f8c67ce74019dca33e749d5c60097ff3 WatchSource:0}: Error finding container eaac51585ffac5c88f8e71cb5b5c4982f8c67ce74019dca33e749d5c60097ff3: Status 404 returned error can't find the container with id eaac51585ffac5c88f8e71cb5b5c4982f8c67ce74019dca33e749d5c60097ff3 Oct 30 05:51:29.195266 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.195220 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f"] Oct 30 05:51:29.515617 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.515568 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" event={"ID":"fc03814b-3458-4fb9-9b32-112310d3649e","Type":"ContainerStarted","Data":"eaac51585ffac5c88f8e71cb5b5c4982f8c67ce74019dca33e749d5c60097ff3"} Oct 30 05:51:29.730727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.730695 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:29.908252 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.908213 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca\") pod \"7366be08-ab4b-4d73-831a-7a0f081d582e\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " Oct 30 05:51:29.908470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.908420 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s\" (UniqueName: \"kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s\") pod \"7366be08-ab4b-4d73-831a-7a0f081d582e\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " Oct 30 05:51:29.908607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.908522 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca\") pod \"7366be08-ab4b-4d73-831a-7a0f081d582e\" (UID: \"7366be08-ab4b-4d73-831a-7a0f081d582e\") " Oct 30 05:51:29.908747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.908716 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca" pod "7366be08-ab4b-4d73-831a-7a0f081d582e" (UID: "7366be08-ab4b-4d73-831a-7a0f081d582e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:29.910066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.909259 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:29.911362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.911334 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca" pod "7366be08-ab4b-4d73-831a-7a0f081d582e" (UID: "7366be08-ab4b-4d73-831a-7a0f081d582e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:29.911475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:29.911364 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s" pod "7366be08-ab4b-4d73-831a-7a0f081d582e" (UID: "7366be08-ab4b-4d73-831a-7a0f081d582e"). InnerVolumeSpecName "kube-api-access-sc76s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:30.010652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.010605 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc76s\" (UniqueName: \"kubernetes.io/projected/7366be08-ab4b-4d73-831a-7a0f081d582e-kube-api-access-sc76s\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:30.010652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.010639 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7366be08-ab4b-4d73-831a-7a0f081d582e-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:30.034332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.034300 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq"] Oct 30 05:51:30.039032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.039003 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-efd5a661-flzcq"] Oct 30 05:51:30.526515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.526484 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" Oct 30 05:51:30.527027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.526484 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx" event={"ID":"7366be08-ab4b-4d73-831a-7a0f081d582e","Type":"ContainerDied","Data":"d558ff46ccbbc627b2ef30986e3256c1f35191c751f0d5fb4c8094e575c0bd7f"} Oct 30 05:51:30.527027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.526631 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d558ff46ccbbc627b2ef30986e3256c1f35191c751f0d5fb4c8094e575c0bd7f" Oct 30 05:51:30.528360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.528337 5565 generic.go:356] "Generic (PLEG): container finished" podID="fc03814b-3458-4fb9-9b32-112310d3649e" containerID="2bf5dcb02289c2306e4c6c182748adb0f96d5a0f7df76babf90ffa38d83adab6" exitCode=0 Oct 30 05:51:30.528478 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:30.528425 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" event={"ID":"fc03814b-3458-4fb9-9b32-112310d3649e","Type":"ContainerDied","Data":"2bf5dcb02289c2306e4c6c182748adb0f96d5a0f7df76babf90ffa38d83adab6"} Oct 30 05:51:31.570409 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.570376 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81bc1317-6af2-4689-99bb-4d696b00d08c" path="/var/lib/kubelet/pods/81bc1317-6af2-4689-99bb-4d696b00d08c/volumes" Oct 30 05:51:31.709863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.709834 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:31.839555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.839426 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca\") pod \"fc03814b-3458-4fb9-9b32-112310d3649e\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " Oct 30 05:51:31.839721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.839608 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q\" (UniqueName: \"kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q\") pod \"fc03814b-3458-4fb9-9b32-112310d3649e\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " Oct 30 05:51:31.839721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.839634 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca\") pod \"fc03814b-3458-4fb9-9b32-112310d3649e\" (UID: \"fc03814b-3458-4fb9-9b32-112310d3649e\") " Oct 30 05:51:31.840084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.840058 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca" pod "fc03814b-3458-4fb9-9b32-112310d3649e" (UID: "fc03814b-3458-4fb9-9b32-112310d3649e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:31.840245 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.840231 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.842126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.842100 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q" pod "fc03814b-3458-4fb9-9b32-112310d3649e" (UID: "fc03814b-3458-4fb9-9b32-112310d3649e"). InnerVolumeSpecName "kube-api-access-4gp6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:31.842209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.842102 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca" pod "fc03814b-3458-4fb9-9b32-112310d3649e" (UID: "fc03814b-3458-4fb9-9b32-112310d3649e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:31.941753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.941718 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gp6q\" (UniqueName: \"kubernetes.io/projected/fc03814b-3458-4fb9-9b32-112310d3649e-kube-api-access-4gp6q\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:31.941753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:31.941748 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03814b-3458-4fb9-9b32-112310d3649e-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:32.032640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:32.032604 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv"] Oct 30 05:51:32.041591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:32.041556 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fe82a2be-9v9rv"] Oct 30 05:51:32.546749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:32.546720 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" Oct 30 05:51:32.546910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:32.546743 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f" event={"ID":"fc03814b-3458-4fb9-9b32-112310d3649e","Type":"ContainerDied","Data":"eaac51585ffac5c88f8e71cb5b5c4982f8c67ce74019dca33e749d5c60097ff3"} Oct 30 05:51:32.546910 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:32.546805 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eaac51585ffac5c88f8e71cb5b5c4982f8c67ce74019dca33e749d5c60097ff3" Oct 30 05:51:33.481899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.481865 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85"] Oct 30 05:51:33.482524 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.482509 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fc03814b-3458-4fb9-9b32-112310d3649e" containerName="revoke-keys" Oct 30 05:51:33.482614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.482557 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc03814b-3458-4fb9-9b32-112310d3649e" containerName="revoke-keys" Oct 30 05:51:33.482614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.482609 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7366be08-ab4b-4d73-831a-7a0f081d582e" containerName="revoke-keys" Oct 30 05:51:33.482678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.482618 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7366be08-ab4b-4d73-831a-7a0f081d582e" containerName="revoke-keys" Oct 30 05:51:33.482771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.482761 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc03814b-3458-4fb9-9b32-112310d3649e" containerName="revoke-keys" Oct 30 05:51:33.482807 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.482778 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7366be08-ab4b-4d73-831a-7a0f081d582e" containerName="revoke-keys" Oct 30 05:51:33.496928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.496905 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.508371 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.508338 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85"] Oct 30 05:51:33.512969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.512945 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:51:33.513238 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.512961 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:33.513369 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.513199 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:33.525106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.525085 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:33.570884 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.570841 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.571087 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.571008 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.571214 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.571199 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.572345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.572322 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c13a086c-9037-492c-8f42-a613c09dfbec" path="/var/lib/kubelet/pods/c13a086c-9037-492c-8f42-a613c09dfbec/volumes" Oct 30 05:51:33.672166 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.672123 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.672354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.672227 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.672441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.672348 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.686707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.686670 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:33.686707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.686697 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:33.692942 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.692910 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.698590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.698570 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.699111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.699080 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw\") pod \"maas-api-revoke-keys-e2e-ait-19e33488-28g85\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:33.830686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.830658 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:51:33.837198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:33.837177 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:34.158144 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:34.158109 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b1afb5_82b7_4ea8_a707_76cb4b21a5f9.slice/crio-5ce33e1898d1a28b1d3848b269a06604c3829b45ae43880d3bd6039700438508 WatchSource:0}: Error finding container 5ce33e1898d1a28b1d3848b269a06604c3829b45ae43880d3bd6039700438508: Status 404 returned error can't find the container with id 5ce33e1898d1a28b1d3848b269a06604c3829b45ae43880d3bd6039700438508 Oct 30 05:51:34.158439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:34.158420 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85"] Oct 30 05:51:34.576044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:34.576004 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" event={"ID":"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9","Type":"ContainerStarted","Data":"5974f16e31741aa11707cd5a2db15f460827d1acc61139ec353cba466032de3e"} Oct 30 05:51:34.576586 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:34.576061 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" event={"ID":"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9","Type":"ContainerStarted","Data":"5ce33e1898d1a28b1d3848b269a06604c3829b45ae43880d3bd6039700438508"} Oct 30 05:51:34.630308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:34.630249 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" podStartSLOduration=1.630234014 podStartE2EDuration="1.630234014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:51:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:51:34.626032404 +0000 UTC m=+1201.201719458" watchObservedRunningTime="2026-10-30 05:51:34.630234014 +0000 UTC m=+1201.205921068" Oct 30 05:51:35.022006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.021973 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp"] Oct 30 05:51:35.034631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.027191 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.050584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.050555 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp"] Oct 30 05:51:35.097595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.094768 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.097595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.094918 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.097595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.094963 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pztqh\" (UniqueName: \"kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.196855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.196802 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.197072 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.196924 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.197072 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.196970 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pztqh\" (UniqueName: \"kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.197686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.197663 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.199784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.199755 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.222618 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.222586 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pztqh\" (UniqueName: \"kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh\") pod \"maas-api-revoke-sub-a95de3f5-l46fp\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.358263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.358159 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:35.585497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.585463 5565 generic.go:356] "Generic (PLEG): container finished" podID="50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" containerID="5974f16e31741aa11707cd5a2db15f460827d1acc61139ec353cba466032de3e" exitCode=0 Oct 30 05:51:35.585899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.585557 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" event={"ID":"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9","Type":"ContainerDied","Data":"5974f16e31741aa11707cd5a2db15f460827d1acc61139ec353cba466032de3e"} Oct 30 05:51:35.677229 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:35.677198 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86712f48_16de_4189_88f8_59f6e2008bba.slice/crio-ca1d28d82fe7d2447eed8ecbb7aec23c8ca4affba989af77ed9bbe3239c69ddd WatchSource:0}: Error finding container ca1d28d82fe7d2447eed8ecbb7aec23c8ca4affba989af77ed9bbe3239c69ddd: Status 404 returned error can't find the container with id ca1d28d82fe7d2447eed8ecbb7aec23c8ca4affba989af77ed9bbe3239c69ddd Oct 30 05:51:35.677570 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:35.677547 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp"] Oct 30 05:51:36.596197 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.596164 5565 generic.go:356] "Generic (PLEG): container finished" podID="86712f48-16de-4189-88f8-59f6e2008bba" containerID="c3f961618ff507cf68f33f336b7415c81439bd95efb866ae4b5b58f80292e6a1" exitCode=0 Oct 30 05:51:36.596630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.596245 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" event={"ID":"86712f48-16de-4189-88f8-59f6e2008bba","Type":"ContainerDied","Data":"c3f961618ff507cf68f33f336b7415c81439bd95efb866ae4b5b58f80292e6a1"} Oct 30 05:51:36.596630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.596309 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" event={"ID":"86712f48-16de-4189-88f8-59f6e2008bba","Type":"ContainerStarted","Data":"ca1d28d82fe7d2447eed8ecbb7aec23c8ca4affba989af77ed9bbe3239c69ddd"} Oct 30 05:51:36.764710 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.764682 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:36.820467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.820435 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca\") pod \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " Oct 30 05:51:36.820666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.820608 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca\") pod \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " Oct 30 05:51:36.820784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.820763 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw\") pod \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\" (UID: \"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9\") " Oct 30 05:51:36.820864 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.820845 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca" pod "50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" (UID: "50b1afb5-82b7-4ea8-a707-76cb4b21a5f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:36.821445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.821417 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:36.823695 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.823665 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw" pod "50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" (UID: "50b1afb5-82b7-4ea8-a707-76cb4b21a5f9"). InnerVolumeSpecName "kube-api-access-xs5bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:36.824667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.824638 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca" pod "50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" (UID: "50b1afb5-82b7-4ea8-a707-76cb4b21a5f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:36.922991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.922917 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:36.922991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:36.922969 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs5bw\" (UniqueName: \"kubernetes.io/projected/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9-kube-api-access-xs5bw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:37.034514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.034472 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx"] Oct 30 05:51:37.040764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.040731 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9d7a17fb-z4mkx"] Oct 30 05:51:37.571946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.571914 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73e71e68-278c-4423-b8e8-a8b7db8d4553" path="/var/lib/kubelet/pods/73e71e68-278c-4423-b8e8-a8b7db8d4553/volumes" Oct 30 05:51:37.607410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.607378 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" Oct 30 05:51:37.607410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.607398 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85" event={"ID":"50b1afb5-82b7-4ea8-a707-76cb4b21a5f9","Type":"ContainerDied","Data":"5ce33e1898d1a28b1d3848b269a06604c3829b45ae43880d3bd6039700438508"} Oct 30 05:51:37.607891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.607449 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ce33e1898d1a28b1d3848b269a06604c3829b45ae43880d3bd6039700438508" Oct 30 05:51:37.638589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.637157 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:51:37.900032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.899998 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:37.938500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.938460 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca\") pod \"86712f48-16de-4189-88f8-59f6e2008bba\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " Oct 30 05:51:37.938708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.938571 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca\") pod \"86712f48-16de-4189-88f8-59f6e2008bba\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " Oct 30 05:51:37.938708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.938677 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh\" (UniqueName: \"kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh\") pod \"86712f48-16de-4189-88f8-59f6e2008bba\" (UID: \"86712f48-16de-4189-88f8-59f6e2008bba\") " Oct 30 05:51:37.940358 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.940326 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca" pod "86712f48-16de-4189-88f8-59f6e2008bba" (UID: "86712f48-16de-4189-88f8-59f6e2008bba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:37.941990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.941953 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca" pod "86712f48-16de-4189-88f8-59f6e2008bba" (UID: "86712f48-16de-4189-88f8-59f6e2008bba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:37.941990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:37.941973 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh" pod "86712f48-16de-4189-88f8-59f6e2008bba" (UID: "86712f48-16de-4189-88f8-59f6e2008bba"). InnerVolumeSpecName "kube-api-access-pztqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:38.041018 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:38.040981 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:38.041018 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:38.041016 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86712f48-16de-4189-88f8-59f6e2008bba-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:38.041229 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:38.041029 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pztqh\" (UniqueName: \"kubernetes.io/projected/86712f48-16de-4189-88f8-59f6e2008bba-kube-api-access-pztqh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:38.617970 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:38.617932 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" event={"ID":"86712f48-16de-4189-88f8-59f6e2008bba","Type":"ContainerDied","Data":"ca1d28d82fe7d2447eed8ecbb7aec23c8ca4affba989af77ed9bbe3239c69ddd"} Oct 30 05:51:38.617970 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:38.617961 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp" Oct 30 05:51:38.618551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:38.617986 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca1d28d82fe7d2447eed8ecbb7aec23c8ca4affba989af77ed9bbe3239c69ddd" Oct 30 05:51:39.042538 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.042492 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s"] Oct 30 05:51:39.043156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.043143 5565 cpu_manager.go:414] "containerMap: removing container" podUID="50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" containerName="revoke-keys" Oct 30 05:51:39.043204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.043158 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" containerName="revoke-keys" Oct 30 05:51:39.043243 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.043207 5565 cpu_manager.go:414] "containerMap: removing container" podUID="86712f48-16de-4189-88f8-59f6e2008bba" containerName="revoke-keys" Oct 30 05:51:39.043243 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.043213 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86712f48-16de-4189-88f8-59f6e2008bba" containerName="revoke-keys" Oct 30 05:51:39.043333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.043324 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="86712f48-16de-4189-88f8-59f6e2008bba" containerName="revoke-keys" Oct 30 05:51:39.043367 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.043339 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" containerName="revoke-keys" Oct 30 05:51:39.046890 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.046863 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.069221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.069192 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s"] Oct 30 05:51:39.074675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.074651 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:51:39.074846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.074734 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:51:39.074846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.074754 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:51:39.074958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.074828 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:51:39.163428 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.163386 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.163647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.163611 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.163734 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.163719 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjs9\" (UniqueName: \"kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.265601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.265561 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjs9\" (UniqueName: \"kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.265785 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.265704 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.265837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.265818 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.266486 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.266453 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.267981 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.267962 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.289989 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.289951 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjs9\" (UniqueName: \"kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9\") pod \"maas-api-revoke-sub-372e855c-f6r7s\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.361264 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.361185 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:39.696799 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:51:39.696754 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf69e09e9_44b7_4474_9f0e_eb93c259fe76.slice/crio-f99366f399dc71b30307e8a96608c72f8f2cba87a6198520199e40ac60e07879 WatchSource:0}: Error finding container f99366f399dc71b30307e8a96608c72f8f2cba87a6198520199e40ac60e07879: Status 404 returned error can't find the container with id f99366f399dc71b30307e8a96608c72f8f2cba87a6198520199e40ac60e07879 Oct 30 05:51:39.698097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:39.698072 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s"] Oct 30 05:51:40.636881 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:40.636847 5565 generic.go:356] "Generic (PLEG): container finished" podID="f69e09e9-44b7-4474-9f0e-eb93c259fe76" containerID="832eacf9cc2642f6712dd9992de3ba6c173cfee40b8fd822a9f3bca8c0051e02" exitCode=0 Oct 30 05:51:40.637058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:40.636892 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" event={"ID":"f69e09e9-44b7-4474-9f0e-eb93c259fe76","Type":"ContainerDied","Data":"832eacf9cc2642f6712dd9992de3ba6c173cfee40b8fd822a9f3bca8c0051e02"} Oct 30 05:51:40.637058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:40.636938 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" event={"ID":"f69e09e9-44b7-4474-9f0e-eb93c259fe76","Type":"ContainerStarted","Data":"f99366f399dc71b30307e8a96608c72f8f2cba87a6198520199e40ac60e07879"} Oct 30 05:51:41.038853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.038812 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s"] Oct 30 05:51:41.044806 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.044760 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bee557b-ddm7s"] Oct 30 05:51:41.569571 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.569513 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09979dbb-f136-4d63-a622-8ddbf72b0d1f" path="/var/lib/kubelet/pods/09979dbb-f136-4d63-a622-8ddbf72b0d1f/volumes" Oct 30 05:51:41.820241 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.820150 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:41.917307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.917263 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca\") pod \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " Oct 30 05:51:41.917545 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.917417 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca\") pod \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " Oct 30 05:51:41.917638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.917553 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9\" (UniqueName: \"kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9\") pod \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\" (UID: \"f69e09e9-44b7-4474-9f0e-eb93c259fe76\") " Oct 30 05:51:41.917822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.917790 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca" pod "f69e09e9-44b7-4474-9f0e-eb93c259fe76" (UID: "f69e09e9-44b7-4474-9f0e-eb93c259fe76"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:41.918571 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.918495 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:41.919852 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.919816 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca" pod "f69e09e9-44b7-4474-9f0e-eb93c259fe76" (UID: "f69e09e9-44b7-4474-9f0e-eb93c259fe76"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:51:41.920661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:41.920631 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9" pod "f69e09e9-44b7-4474-9f0e-eb93c259fe76" (UID: "f69e09e9-44b7-4474-9f0e-eb93c259fe76"). InnerVolumeSpecName "kube-api-access-gcjs9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:42.020017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:42.019979 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f69e09e9-44b7-4474-9f0e-eb93c259fe76-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:42.020017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:42.020014 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjs9\" (UniqueName: \"kubernetes.io/projected/f69e09e9-44b7-4474-9f0e-eb93c259fe76-kube-api-access-gcjs9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:42.659648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:42.659617 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" Oct 30 05:51:42.660106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:42.659650 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s" event={"ID":"f69e09e9-44b7-4474-9f0e-eb93c259fe76","Type":"ContainerDied","Data":"f99366f399dc71b30307e8a96608c72f8f2cba87a6198520199e40ac60e07879"} Oct 30 05:51:42.660106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:42.659712 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f99366f399dc71b30307e8a96608c72f8f2cba87a6198520199e40ac60e07879" Oct 30 05:51:43.037647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:43.037601 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg"] Oct 30 05:51:43.044453 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:43.044415 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8851028d-vdzmg"] Oct 30 05:51:43.571280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:43.571243 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1300fbbc-7ab5-47fb-825a-3384ff24a461" path="/var/lib/kubelet/pods/1300fbbc-7ab5-47fb-825a-3384ff24a461/volumes" Oct 30 05:51:44.716642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:44.716498 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976"] Oct 30 05:51:44.754098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:44.754049 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5"] Oct 30 05:51:47.533755 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:51:47.533715 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:51:49.717168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:49.717128 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" podUID="3102109e-9eb0-479f-8b72-12291b0305ed" containerName="payload-processing" containerID="cri-o://41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0" gracePeriod=25 Oct 30 05:51:49.754665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:49.754630 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" podUID="8e9f6598-fb41-4f5a-833f-d705aec48459" containerName="payload-pre-processing" containerID="cri-o://64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b" gracePeriod=25 Oct 30 05:51:50.033159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.033127 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:51:50.047837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.047810 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k\" (UniqueName: \"kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k\") pod \"3102109e-9eb0-479f-8b72-12291b0305ed\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " Oct 30 05:51:50.047991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.047922 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume\") pod \"3102109e-9eb0-479f-8b72-12291b0305ed\" (UID: \"3102109e-9eb0-479f-8b72-12291b0305ed\") " Oct 30 05:51:50.048253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.048229 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume" pod "3102109e-9eb0-479f-8b72-12291b0305ed" (UID: "3102109e-9eb0-479f-8b72-12291b0305ed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:50.048671 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.048651 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3102109e-9eb0-479f-8b72-12291b0305ed-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:50.049976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.049952 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k" pod "3102109e-9eb0-479f-8b72-12291b0305ed" (UID: "3102109e-9eb0-479f-8b72-12291b0305ed"). InnerVolumeSpecName "kube-api-access-gc72k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:50.104180 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.104130 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:51:50.169239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.169197 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume\") pod \"8e9f6598-fb41-4f5a-833f-d705aec48459\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " Oct 30 05:51:50.169429 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.169409 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd\" (UniqueName: \"kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd\") pod \"8e9f6598-fb41-4f5a-833f-d705aec48459\" (UID: \"8e9f6598-fb41-4f5a-833f-d705aec48459\") " Oct 30 05:51:50.170688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.170662 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gc72k\" (UniqueName: \"kubernetes.io/projected/3102109e-9eb0-479f-8b72-12291b0305ed-kube-api-access-gc72k\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:50.173104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.173078 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume" pod "8e9f6598-fb41-4f5a-833f-d705aec48459" (UID: "8e9f6598-fb41-4f5a-833f-d705aec48459"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:51:50.173504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.173466 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd" pod "8e9f6598-fb41-4f5a-833f-d705aec48459" (UID: "8e9f6598-fb41-4f5a-833f-d705aec48459"). InnerVolumeSpecName "kube-api-access-945nd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:51:50.272988 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.272949 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e9f6598-fb41-4f5a-833f-d705aec48459-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:50.272988 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.272989 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-945nd\" (UniqueName: \"kubernetes.io/projected/8e9f6598-fb41-4f5a-833f-d705aec48459-kube-api-access-945nd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:51:50.739435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.739400 5565 generic.go:356] "Generic (PLEG): container finished" podID="3102109e-9eb0-479f-8b72-12291b0305ed" containerID="41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0" exitCode=0 Oct 30 05:51:50.739814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.739467 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" Oct 30 05:51:50.739814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.739479 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" event={"ID":"3102109e-9eb0-479f-8b72-12291b0305ed","Type":"ContainerDied","Data":"41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0"} Oct 30 05:51:50.739814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.739549 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976" event={"ID":"3102109e-9eb0-479f-8b72-12291b0305ed","Type":"ContainerDied","Data":"0958c082b6316236f20ec56b0bc4aecf37903ba0e40533603856a15950559edc"} Oct 30 05:51:50.739814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.739572 5565 scope.go:122] "RemoveContainer" containerID="41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0" Oct 30 05:51:50.741374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.741353 5565 generic.go:356] "Generic (PLEG): container finished" podID="8e9f6598-fb41-4f5a-833f-d705aec48459" containerID="64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b" exitCode=0 Oct 30 05:51:50.741479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.741416 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" Oct 30 05:51:50.741479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.741439 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" event={"ID":"8e9f6598-fb41-4f5a-833f-d705aec48459","Type":"ContainerDied","Data":"64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b"} Oct 30 05:51:50.741597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.741481 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5" event={"ID":"8e9f6598-fb41-4f5a-833f-d705aec48459","Type":"ContainerDied","Data":"e92ec379b7ca884ba36b1d4c6b88b94bcdc2b9c58286c511aab432e4d97a51d5"} Oct 30 05:51:50.752724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.752708 5565 scope.go:122] "RemoveContainer" containerID="41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0" Oct 30 05:51:50.753008 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:51:50.752990 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0\": container with ID starting with 41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0 not found: ID does not exist" containerID="41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0" Oct 30 05:51:50.753069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.753016 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0"} err="failed to get container status \"41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0\": rpc error: code = NotFound desc = could not find container \"41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0\": container with ID starting with 41b5e9b6f2018097ac9e4c2455e86159fdea5caf246b1f1698aef0838ab02ec0 not found: ID does not exist" Oct 30 05:51:50.753069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.753033 5565 scope.go:122] "RemoveContainer" containerID="64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b" Oct 30 05:51:50.764661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.764644 5565 scope.go:122] "RemoveContainer" containerID="64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b" Oct 30 05:51:50.764943 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:51:50.764926 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b\": container with ID starting with 64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b not found: ID does not exist" containerID="64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b" Oct 30 05:51:50.765007 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.764952 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b"} err="failed to get container status \"64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b\": rpc error: code = NotFound desc = could not find container \"64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b\": container with ID starting with 64445bf019cb1ae6031be9bb7fafdd5403800e3150498bec1b0c0de9760f2d5b not found: ID does not exist" Oct 30 05:51:50.800098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.800073 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976"] Oct 30 05:51:50.805207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.805182 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-19e33488-5798dcbf4d-zg976"] Oct 30 05:51:50.839870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.839845 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5"] Oct 30 05:51:50.847757 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:50.847734 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-19e33488-5cb656d495-5wlz5"] Oct 30 05:51:51.568009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:51.567978 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3102109e-9eb0-479f-8b72-12291b0305ed" path="/var/lib/kubelet/pods/3102109e-9eb0-479f-8b72-12291b0305ed/volumes" Oct 30 05:51:51.568430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:51.568415 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e9f6598-fb41-4f5a-833f-d705aec48459" path="/var/lib/kubelet/pods/8e9f6598-fb41-4f5a-833f-d705aec48459/volumes" Oct 30 05:51:54.080338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:54.080303 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9"] Oct 30 05:51:54.093360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:54.093328 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd"] Oct 30 05:51:54.098169 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:54.098145 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-502017e5-cdmj9"] Oct 30 05:51:54.102135 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:54.102113 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-274cd53a-f8chd"] Oct 30 05:51:55.573343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:55.573311 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d246ee08-db83-42f1-a7ca-e2fefd0c004a" path="/var/lib/kubelet/pods/d246ee08-db83-42f1-a7ca-e2fefd0c004a/volumes" Oct 30 05:51:55.573868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:51:55.573831 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe5f6735-b306-4bc1-af35-741411c0b65e" path="/var/lib/kubelet/pods/fe5f6735-b306-4bc1-af35-741411c0b65e/volumes" Oct 30 05:52:01.694119 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:01.694080 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85"] Oct 30 05:52:01.706447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:01.706417 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-19e33488-28g85"] Oct 30 05:52:02.523487 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.523451 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t"] Oct 30 05:52:02.524221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524202 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f69e09e9-44b7-4474-9f0e-eb93c259fe76" containerName="revoke-keys" Oct 30 05:52:02.524299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524222 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f69e09e9-44b7-4474-9f0e-eb93c259fe76" containerName="revoke-keys" Oct 30 05:52:02.524299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524238 5565 cpu_manager.go:414] "containerMap: removing container" podUID="8e9f6598-fb41-4f5a-833f-d705aec48459" containerName="payload-pre-processing" Oct 30 05:52:02.524299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524244 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e9f6598-fb41-4f5a-833f-d705aec48459" containerName="payload-pre-processing" Oct 30 05:52:02.524299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524264 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3102109e-9eb0-479f-8b72-12291b0305ed" containerName="payload-processing" Oct 30 05:52:02.524299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524269 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3102109e-9eb0-479f-8b72-12291b0305ed" containerName="payload-processing" Oct 30 05:52:02.524458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524394 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f69e09e9-44b7-4474-9f0e-eb93c259fe76" containerName="revoke-keys" Oct 30 05:52:02.524458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524402 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e9f6598-fb41-4f5a-833f-d705aec48459" containerName="payload-pre-processing" Oct 30 05:52:02.524458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.524412 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3102109e-9eb0-479f-8b72-12291b0305ed" containerName="payload-processing" Oct 30 05:52:02.529228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.529208 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.548747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.548718 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t"] Oct 30 05:52:02.556000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.555969 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:52:02.556224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.556203 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:52:02.616744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.616716 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:52:02.616999 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.616984 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:52:02.659108 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.659082 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.659283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.659262 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.659489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.659472 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b799g\" (UniqueName: \"kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.760860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.760835 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.761209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.760898 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.761209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.760968 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b799g\" (UniqueName: \"kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.761481 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.761463 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.763154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.763135 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.799241 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.799183 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b799g\" (UniqueName: \"kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g\") pod \"maas-api-revoke-keys-e2e-derive-3224b039-9fz2t\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:02.843748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:02.843708 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:03.156288 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:52:03.156251 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1549a2a_b145_4bf7_ad5a_005e003c6108.slice/crio-5dd4645aa61ce835441528b1e2135fd6bcf6d71cf156690c66a1d6987c9d6e44 WatchSource:0}: Error finding container 5dd4645aa61ce835441528b1e2135fd6bcf6d71cf156690c66a1d6987c9d6e44: Status 404 returned error can't find the container with id 5dd4645aa61ce835441528b1e2135fd6bcf6d71cf156690c66a1d6987c9d6e44 Oct 30 05:52:03.160928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:03.159520 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t"] Oct 30 05:52:03.568587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:03.568556 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50b1afb5-82b7-4ea8-a707-76cb4b21a5f9" path="/var/lib/kubelet/pods/50b1afb5-82b7-4ea8-a707-76cb4b21a5f9/volumes" Oct 30 05:52:03.876405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:03.876307 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" event={"ID":"c1549a2a-b145-4bf7-ad5a-005e003c6108","Type":"ContainerStarted","Data":"be46412fad008b7198ff0823d55add2557c5f20e51dd3fb913317f9212a72370"} Oct 30 05:52:03.876405 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:03.876364 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" event={"ID":"c1549a2a-b145-4bf7-ad5a-005e003c6108","Type":"ContainerStarted","Data":"5dd4645aa61ce835441528b1e2135fd6bcf6d71cf156690c66a1d6987c9d6e44"} Oct 30 05:52:03.926733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:03.926681 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" podStartSLOduration=1.926666845 podStartE2EDuration="1.926666845s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:03.925298519 +0000 UTC m=+1230.500985574" watchObservedRunningTime="2026-10-30 05:52:03.926666845 +0000 UTC m=+1230.502353901" Oct 30 05:52:04.885867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:04.885835 5565 generic.go:356] "Generic (PLEG): container finished" podID="c1549a2a-b145-4bf7-ad5a-005e003c6108" containerID="be46412fad008b7198ff0823d55add2557c5f20e51dd3fb913317f9212a72370" exitCode=0 Oct 30 05:52:04.886240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:04.885919 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" event={"ID":"c1549a2a-b145-4bf7-ad5a-005e003c6108","Type":"ContainerDied","Data":"be46412fad008b7198ff0823d55add2557c5f20e51dd3fb913317f9212a72370"} Oct 30 05:52:05.161227 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:05.161137 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:52:06.082715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.082692 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:06.111545 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.111494 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca\") pod \"c1549a2a-b145-4bf7-ad5a-005e003c6108\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " Oct 30 05:52:06.111715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.111578 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca\") pod \"c1549a2a-b145-4bf7-ad5a-005e003c6108\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " Oct 30 05:52:06.111715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.111699 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g\" (UniqueName: \"kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g\") pod \"c1549a2a-b145-4bf7-ad5a-005e003c6108\" (UID: \"c1549a2a-b145-4bf7-ad5a-005e003c6108\") " Oct 30 05:52:06.111962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.111930 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca" pod "c1549a2a-b145-4bf7-ad5a-005e003c6108" (UID: "c1549a2a-b145-4bf7-ad5a-005e003c6108"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:06.112386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.112362 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:06.114213 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.114189 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca" pod "c1549a2a-b145-4bf7-ad5a-005e003c6108" (UID: "c1549a2a-b145-4bf7-ad5a-005e003c6108"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:06.114213 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.114201 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g" pod "c1549a2a-b145-4bf7-ad5a-005e003c6108" (UID: "c1549a2a-b145-4bf7-ad5a-005e003c6108"). InnerVolumeSpecName "kube-api-access-b799g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:06.215621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.213940 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1549a2a-b145-4bf7-ad5a-005e003c6108-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:06.215621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.213976 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b799g\" (UniqueName: \"kubernetes.io/projected/c1549a2a-b145-4bf7-ad5a-005e003c6108-kube-api-access-b799g\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:06.908782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.908749 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" event={"ID":"c1549a2a-b145-4bf7-ad5a-005e003c6108","Type":"ContainerDied","Data":"5dd4645aa61ce835441528b1e2135fd6bcf6d71cf156690c66a1d6987c9d6e44"} Oct 30 05:52:06.909010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.908999 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5dd4645aa61ce835441528b1e2135fd6bcf6d71cf156690c66a1d6987c9d6e44" Oct 30 05:52:06.909126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:06.909118 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t" Oct 30 05:52:07.735337 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.735296 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c"] Oct 30 05:52:07.737124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.737080 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c1549a2a-b145-4bf7-ad5a-005e003c6108" containerName="revoke-keys" Oct 30 05:52:07.737124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.737119 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1549a2a-b145-4bf7-ad5a-005e003c6108" containerName="revoke-keys" Oct 30 05:52:07.737339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.737325 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1549a2a-b145-4bf7-ad5a-005e003c6108" containerName="revoke-keys" Oct 30 05:52:07.746668 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.746615 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.773849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.763724 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msmmt\" (UniqueName: \"kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.773849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.773137 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.773849 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.773401 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.791112 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.791072 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:52:07.791575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.791549 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:52:07.791879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.791860 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:52:07.800002 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.792119 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:52:07.837993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.837944 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c"] Oct 30 05:52:07.881674 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.881564 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.881878 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.881782 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.881971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.881951 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msmmt\" (UniqueName: \"kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.883163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.883138 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.885773 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.885746 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:07.934810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:07.934631 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-msmmt\" (UniqueName: \"kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt\") pod \"maas-api-revoke-sub-318cd5c3-wsh4c\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:08.139155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:08.139118 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:08.561657 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:08.561623 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c"] Oct 30 05:52:08.562636 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:52:08.562608 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39dcf4f4_6dd1_4a7a_8f11_4ff3c26af149.slice/crio-9837226ba0869ad60c197cdd3edde5b678fb559f1f94c7cd762d5b4254a55ec1 WatchSource:0}: Error finding container 9837226ba0869ad60c197cdd3edde5b678fb559f1f94c7cd762d5b4254a55ec1: Status 404 returned error can't find the container with id 9837226ba0869ad60c197cdd3edde5b678fb559f1f94c7cd762d5b4254a55ec1 Oct 30 05:52:08.945042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:08.944955 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" event={"ID":"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149","Type":"ContainerStarted","Data":"9837226ba0869ad60c197cdd3edde5b678fb559f1f94c7cd762d5b4254a55ec1"} Oct 30 05:52:09.955870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:09.955835 5565 generic.go:356] "Generic (PLEG): container finished" podID="39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" containerID="9ac06da60fb9945f049e2f445e5e726e6ff290f320b67930a758546da596651d" exitCode=0 Oct 30 05:52:09.956327 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:09.955919 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" event={"ID":"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149","Type":"ContainerDied","Data":"9ac06da60fb9945f049e2f445e5e726e6ff290f320b67930a758546da596651d"} Oct 30 05:52:11.154934 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.154903 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:11.272235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.272196 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca\") pod \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " Oct 30 05:52:11.272439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.272346 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca\") pod \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " Oct 30 05:52:11.272597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.272578 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt\" (UniqueName: \"kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt\") pod \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\" (UID: \"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149\") " Oct 30 05:52:11.272812 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.272782 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca" pod "39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" (UID: "39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:11.273265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.273240 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:11.274751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.274721 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca" pod "39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" (UID: "39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:52:11.274874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.274759 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt" pod "39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" (UID: "39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149"). InnerVolumeSpecName "kube-api-access-msmmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:11.374744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.374705 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-msmmt\" (UniqueName: \"kubernetes.io/projected/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-kube-api-access-msmmt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:11.374744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.374735 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:11.977865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.977782 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" Oct 30 05:52:11.977865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.977810 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c" event={"ID":"39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149","Type":"ContainerDied","Data":"9837226ba0869ad60c197cdd3edde5b678fb559f1f94c7cd762d5b4254a55ec1"} Oct 30 05:52:11.978061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:11.977885 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9837226ba0869ad60c197cdd3edde5b678fb559f1f94c7cd762d5b4254a55ec1" Oct 30 05:52:17.537521 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:52:17.537479 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:52:17.609365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:17.609329 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t"] Oct 30 05:52:17.620117 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:17.620093 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-3224b039-9fz2t"] Oct 30 05:52:19.568589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:19.568555 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1549a2a-b145-4bf7-ad5a-005e003c6108" path="/var/lib/kubelet/pods/c1549a2a-b145-4bf7-ad5a-005e003c6108/volumes" Oct 30 05:52:21.894850 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.894819 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw"] Oct 30 05:52:21.895514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.895496 5565 cpu_manager.go:414] "containerMap: removing container" podUID="39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" containerName="revoke-keys" Oct 30 05:52:21.895594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.895516 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" containerName="revoke-keys" Oct 30 05:52:21.895694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.895682 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" containerName="revoke-keys" Oct 30 05:52:21.901281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.901263 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:21.922370 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.922342 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw"] Oct 30 05:52:21.928948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.928918 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-0b481b\"" Oct 30 05:52:21.929070 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.928925 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-0b481b-dockercfg-tsrdk\"" Oct 30 05:52:21.941779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.941755 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsm5z\" (UniqueName: \"kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z\") pod \"payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:21.941891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.941824 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:21.983467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.983428 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7"] Oct 30 05:52:21.989860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:21.989835 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.038554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.029559 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7"] Oct 30 05:52:22.085280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.085238 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsm5z\" (UniqueName: \"kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z\") pod \"payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:22.086738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.086705 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:22.088069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.088039 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:22.117231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.117149 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsm5z\" (UniqueName: \"kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z\") pod \"payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:22.188684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.188604 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxg9w\" (UniqueName: \"kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w\") pod \"payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.188971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.188948 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.216159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.216128 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:22.291653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.291211 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxg9w\" (UniqueName: \"kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w\") pod \"payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.291784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.291708 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.294683 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.293294 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.316308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.316206 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxg9w\" (UniqueName: \"kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w\") pod \"payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.457865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.457782 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:22.592626 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.592032 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw"] Oct 30 05:52:22.844862 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:52:22.844821 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fda9767_1e03_4e98_800f_e9ed34dedf47.slice/crio-2a0ac6f8783e3b07ebeebe7e1b125fc7b730c1110534e64e59d90663fc506e7b WatchSource:0}: Error finding container 2a0ac6f8783e3b07ebeebe7e1b125fc7b730c1110534e64e59d90663fc506e7b: Status 404 returned error can't find the container with id 2a0ac6f8783e3b07ebeebe7e1b125fc7b730c1110534e64e59d90663fc506e7b Oct 30 05:52:22.846567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:22.846481 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7"] Oct 30 05:52:23.171711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.171619 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" event={"ID":"81e69693-69c0-4d07-867e-8d385f41c342","Type":"ContainerStarted","Data":"434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791"} Oct 30 05:52:23.171711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.171683 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" event={"ID":"81e69693-69c0-4d07-867e-8d385f41c342","Type":"ContainerStarted","Data":"c4aa5c1e3202d8deb167eb5d82a11eadfc7b07f9ac8175dfa920abdf785bb528"} Oct 30 05:52:23.171711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.171712 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:23.173455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.173429 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" event={"ID":"9fda9767-1e03-4e98-800f-e9ed34dedf47","Type":"ContainerStarted","Data":"566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25"} Oct 30 05:52:23.173623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.173478 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" event={"ID":"9fda9767-1e03-4e98-800f-e9ed34dedf47","Type":"ContainerStarted","Data":"2a0ac6f8783e3b07ebeebe7e1b125fc7b730c1110534e64e59d90663fc506e7b"} Oct 30 05:52:23.173623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.173508 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:23.263365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.263316 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" podStartSLOduration=2.26330192 podStartE2EDuration="2.26330192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:23.218682517 +0000 UTC m=+1249.794369571" watchObservedRunningTime="2026-10-30 05:52:23.26330192 +0000 UTC m=+1249.838988973" Oct 30 05:52:23.264071 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:23.264039 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" podStartSLOduration=2.264029139 podStartE2EDuration="2.264029139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:23.259657154 +0000 UTC m=+1249.835344209" watchObservedRunningTime="2026-10-30 05:52:23.264029139 +0000 UTC m=+1249.839716194" Oct 30 05:52:25.995871 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:25.995836 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw"] Oct 30 05:52:26.000847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.000826 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.017946 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.017922 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-8bab43d7\"" Oct 30 05:52:26.023868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.023835 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54"] Oct 30 05:52:26.028980 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.028956 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.030120 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.030092 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw"] Oct 30 05:52:26.050469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.050416 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-8bab43d7-dockercfg-z9fnq\"" Oct 30 05:52:26.059543 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.059498 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjhg8\" (UniqueName: \"kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8\") pod \"payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.059735 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.059709 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.059885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.059865 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume\") pod \"payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.059923 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.059896 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff5vj\" (UniqueName: \"kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj\") pod \"payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.063709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.063686 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54"] Oct 30 05:52:26.170095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.168838 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjhg8\" (UniqueName: \"kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8\") pod \"payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.170095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.168984 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.170095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.169166 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume\") pod \"payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.170095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.169199 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff5vj\" (UniqueName: \"kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj\") pod \"payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.172997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.171058 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.172997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.171073 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume\") pod \"payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.206425 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.206392 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff5vj\" (UniqueName: \"kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj\") pod \"payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.210451 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.210188 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjhg8\" (UniqueName: \"kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8\") pod \"payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.320207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.320169 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:26.359296 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.359255 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:26.646303 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:52:26.646264 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaae66a6_23c5_49f8_96a0_ccb809394308.slice/crio-f51911a2df2e9143db5a716cb84e7ba6d9523cec4093ec2226728ac204343431 WatchSource:0}: Error finding container f51911a2df2e9143db5a716cb84e7ba6d9523cec4093ec2226728ac204343431: Status 404 returned error can't find the container with id f51911a2df2e9143db5a716cb84e7ba6d9523cec4093ec2226728ac204343431 Oct 30 05:52:26.647441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.647416 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw"] Oct 30 05:52:26.695742 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:52:26.695634 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod676d796e_7f82_43f3_8fc6_9207c20f4d36.slice/crio-a3f0ce1399286596f57372610f6a2e6dab18ed720509d2afc821d3702df4dcb2 WatchSource:0}: Error finding container a3f0ce1399286596f57372610f6a2e6dab18ed720509d2afc821d3702df4dcb2: Status 404 returned error can't find the container with id a3f0ce1399286596f57372610f6a2e6dab18ed720509d2afc821d3702df4dcb2 Oct 30 05:52:26.697237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:26.697200 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54"] Oct 30 05:52:27.258656 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.258619 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" event={"ID":"faae66a6-23c5-49f8-96a0-ccb809394308","Type":"ContainerStarted","Data":"9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa"} Oct 30 05:52:27.259092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.258712 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:27.259092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.258741 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" event={"ID":"faae66a6-23c5-49f8-96a0-ccb809394308","Type":"ContainerStarted","Data":"f51911a2df2e9143db5a716cb84e7ba6d9523cec4093ec2226728ac204343431"} Oct 30 05:52:27.260611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.260555 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" event={"ID":"676d796e-7f82-43f3-8fc6-9207c20f4d36","Type":"ContainerStarted","Data":"34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac"} Oct 30 05:52:27.260611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.260607 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" event={"ID":"676d796e-7f82-43f3-8fc6-9207c20f4d36","Type":"ContainerStarted","Data":"a3f0ce1399286596f57372610f6a2e6dab18ed720509d2afc821d3702df4dcb2"} Oct 30 05:52:27.260811 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.260633 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:27.308435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.308376 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" podStartSLOduration=2.30836212 podStartE2EDuration="2.30836212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:27.306055372 +0000 UTC m=+1253.881742425" watchObservedRunningTime="2026-10-30 05:52:27.30836212 +0000 UTC m=+1253.884049218" Oct 30 05:52:27.372356 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:27.372300 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" podStartSLOduration=2.372287054 podStartE2EDuration="2.372287054s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:27.369179893 +0000 UTC m=+1253.944866948" watchObservedRunningTime="2026-10-30 05:52:27.372287054 +0000 UTC m=+1253.947974102" Oct 30 05:52:29.089208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.089134 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4"] Oct 30 05:52:29.094829 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.094802 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.116430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.116397 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4"] Oct 30 05:52:29.122421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.122395 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d96a1f\"" Oct 30 05:52:29.123136 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.123107 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.123261 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.123234 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mkv\" (UniqueName: \"kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv\") pod \"payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.123427 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.122377 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d96a1f-dockercfg-9qkvg\"" Oct 30 05:52:29.226547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.224936 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mkv\" (UniqueName: \"kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv\") pod \"payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.226547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.225287 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.240624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.231835 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.320292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.320230 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mkv\" (UniqueName: \"kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv\") pod \"payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.415227 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.415149 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:29.748690 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:52:29.748653 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafa4b166_c592_4836_b7fc_d5548a148676.slice/crio-812fe9852fe0fe29a8a3cc704f473528e8de72b9a5f3c8f871c23f98cbd145a6 WatchSource:0}: Error finding container 812fe9852fe0fe29a8a3cc704f473528e8de72b9a5f3c8f871c23f98cbd145a6: Status 404 returned error can't find the container with id 812fe9852fe0fe29a8a3cc704f473528e8de72b9a5f3c8f871c23f98cbd145a6 Oct 30 05:52:29.749796 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:29.749774 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4"] Oct 30 05:52:30.340664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:30.340625 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" event={"ID":"afa4b166-c592-4836-b7fc-d5548a148676","Type":"ContainerStarted","Data":"c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467"} Oct 30 05:52:30.341061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:30.340730 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" event={"ID":"afa4b166-c592-4836-b7fc-d5548a148676","Type":"ContainerStarted","Data":"812fe9852fe0fe29a8a3cc704f473528e8de72b9a5f3c8f871c23f98cbd145a6"} Oct 30 05:52:30.341061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:30.340872 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:30.391667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:30.391586 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" podStartSLOduration=1.391567017 podStartE2EDuration="1.391567017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:52:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:52:30.385945842 +0000 UTC m=+1256.961632897" watchObservedRunningTime="2026-10-30 05:52:30.391567017 +0000 UTC m=+1256.967254070" Oct 30 05:52:33.858579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.858541 5565 scope.go:122] "RemoveContainer" containerID="3e9add7862d969138dbc42f0b37677e37d67f0f59a1baa7dacbbcc0d6111fa6b" Oct 30 05:52:33.875926 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.875901 5565 scope.go:122] "RemoveContainer" containerID="1ad8fa516781fe87a8e83c2b137b6dc41707682433211af76575b01759a626f3" Oct 30 05:52:33.890266 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.889740 5565 scope.go:122] "RemoveContainer" containerID="58beee501640ff50d7aaee9868dd6e3785765a52b39a56121b06cf0f7244cae7" Oct 30 05:52:33.900374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.900352 5565 scope.go:122] "RemoveContainer" containerID="252ecb187a2a849e72cb5b28253d0817ae70ec96a0b41b2d67bc7011a6daf2d1" Oct 30 05:52:33.910408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.910387 5565 scope.go:122] "RemoveContainer" containerID="092973f40dbb89d99b035cdbd1f0fb9e99c912aa0c1e9c607acba796364315e4" Oct 30 05:52:33.919698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.919679 5565 scope.go:122] "RemoveContainer" containerID="3e43a916784ee1f27301d640534e15ce1303db6a32e4af6fac62edbddf93b6c0" Oct 30 05:52:33.928828 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.928812 5565 scope.go:122] "RemoveContainer" containerID="f4e2af4a4d5b3408b37977cdfb3dfbdd9ebff78a1168974c5e6b30dbea271856" Oct 30 05:52:33.938114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.938097 5565 scope.go:122] "RemoveContainer" containerID="70a6e4707e8ef50c114433011f8d8fdac108488c77d0d5dd0123aaa417e5b92d" Oct 30 05:52:33.946993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:33.946975 5565 scope.go:122] "RemoveContainer" containerID="ea02ab23d50b5e175f9ffd2eba9e6e1f6b70d667ff74a1580457bb108ebd31ba" Oct 30 05:52:34.185167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:34.185090 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 05:52:34.185167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:34.185153 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 05:52:34.785419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:34.785373 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv"] Oct 30 05:52:34.785704 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:34.785675 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" containerID="cri-o://4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58" gracePeriod=30 Oct 30 05:52:38.276987 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:38.272909 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:52:38.276987 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:38.273028 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:52:40.040379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.040350 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp"] Oct 30 05:52:40.049579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.049553 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7335ba6e-5vrjp"] Oct 30 05:52:40.069192 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.069171 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:52:40.201392 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201353 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201485 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201559 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201601 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201624 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201671 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:40.201883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201806 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201844 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201894 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:52:40.201985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201932 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.201985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.201969 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token\") pod \"1711d7c4-786c-436a-bea7-df643516fdcc\" (UID: \"1711d7c4-786c-436a-bea7-df643516fdcc\") " Oct 30 05:52:40.202143 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.202079 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:40.202240 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.202223 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:40.202745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.202721 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.202745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.202747 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1711d7c4-786c-436a-bea7-df643516fdcc-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.203020 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.202762 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.203020 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.202775 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.203341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.203319 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:40.204709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.204683 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:40.204822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.204727 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:52:40.205093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.205070 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "kube-api-access-x2mzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:52:40.205289 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.205271 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy" pod "1711d7c4-786c-436a-bea7-df643516fdcc" (UID: "1711d7c4-786c-436a-bea7-df643516fdcc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:52:40.304515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.304417 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.304515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.304446 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.304515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.304457 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1711d7c4-786c-436a-bea7-df643516fdcc-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.304515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.304466 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x2mzr\" (UniqueName: \"kubernetes.io/projected/1711d7c4-786c-436a-bea7-df643516fdcc-kube-api-access-x2mzr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.304515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.304476 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1711d7c4-786c-436a-bea7-df643516fdcc-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:52:40.436942 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.436901 5565 generic.go:356] "Generic (PLEG): container finished" podID="1711d7c4-786c-436a-bea7-df643516fdcc" containerID="4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58" exitCode=0 Oct 30 05:52:40.437125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.436962 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" Oct 30 05:52:40.437125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.436973 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" event={"ID":"1711d7c4-786c-436a-bea7-df643516fdcc","Type":"ContainerDied","Data":"4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58"} Oct 30 05:52:40.437125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.437044 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv" event={"ID":"1711d7c4-786c-436a-bea7-df643516fdcc","Type":"ContainerDied","Data":"1d269bb8fba9444aa097617a29c2e38204c5aabf2cc1a92fc9a573042ae3336c"} Oct 30 05:52:40.437125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.437079 5565 scope.go:122] "RemoveContainer" containerID="4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58" Oct 30 05:52:40.450657 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.450638 5565 scope.go:122] "RemoveContainer" containerID="4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58" Oct 30 05:52:40.451009 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:52:40.450983 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58\": container with ID starting with 4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58 not found: ID does not exist" containerID="4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58" Oct 30 05:52:40.451126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.451017 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58"} err="failed to get container status \"4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58\": rpc error: code = NotFound desc = could not find container \"4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58\": container with ID starting with 4d11fae7bdfb8084a984dfec3eb6fa2da259257b7a062cbcb6ef2b1234483a58 not found: ID does not exist" Oct 30 05:52:40.495059 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.495039 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv"] Oct 30 05:52:40.499640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:40.499621 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-9b5mv"] Oct 30 05:52:41.353876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:41.353850 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 05:52:41.580816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:41.580738 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" path="/var/lib/kubelet/pods/1711d7c4-786c-436a-bea7-df643516fdcc/volumes" Oct 30 05:52:41.581798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:41.581773 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4395bb69-11d9-4aaf-932a-37c233f8861f" path="/var/lib/kubelet/pods/4395bb69-11d9-4aaf-932a-37c233f8861f/volumes" Oct 30 05:52:47.532698 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:52:47.532612 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:52:58.034147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:58.034105 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h"] Oct 30 05:52:58.041775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:58.041734 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8efe04be-sh49h"] Oct 30 05:52:59.569161 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:52:59.569126 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53f57850-b05c-476c-a1c8-4737a072ed55" path="/var/lib/kubelet/pods/53f57850-b05c-476c-a1c8-4737a072ed55/volumes" Oct 30 05:53:01.474801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.474766 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8"] Oct 30 05:53:01.475762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.475740 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" Oct 30 05:53:01.475762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.475761 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" Oct 30 05:53:01.475957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.475944 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1711d7c4-786c-436a-bea7-df643516fdcc" containerName="istio-proxy" Oct 30 05:53:01.481449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.481429 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.507594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.507565 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-b4ca14-openshift-default-dockercfg-fjppn\"" Oct 30 05:53:01.524993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.524966 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8"] Oct 30 05:53:01.579940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.579907 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580091 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.579955 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580091 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580037 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580091 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580083 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580206 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580189 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580214 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580289 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580261 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580330 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqmgf\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.580383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.580368 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.684332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.684300 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.684705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.684676 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.684868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.684852 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.685056 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685038 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685199 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqmgf\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685271 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685438 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685446 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685502 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685578 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.685932 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.686118 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.687128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.687058 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.702565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.690778 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.726157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.725209 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.726157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.725291 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.751064 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.740453 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqmgf\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.752015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.751986 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token\") pod \"e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:01.798407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:01.798371 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:02.131421 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:02.131389 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d65898f_a620_49fa_8930_eba9e5e61287.slice/crio-1c2f0e689e332a5e572e63049dee8655a77972b916fa15ef5e794fba5123486b WatchSource:0}: Error finding container 1c2f0e689e332a5e572e63049dee8655a77972b916fa15ef5e794fba5123486b: Status 404 returned error can't find the container with id 1c2f0e689e332a5e572e63049dee8655a77972b916fa15ef5e794fba5123486b Oct 30 05:53:02.132891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.132864 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8"] Oct 30 05:53:02.134140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.134108 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:02.134222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.134173 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:02.134222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.134201 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:02.737075 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.737036 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" event={"ID":"2d65898f-a620-49fa-8930-eba9e5e61287","Type":"ContainerStarted","Data":"35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927"} Oct 30 05:53:02.737420 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.737097 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" event={"ID":"2d65898f-a620-49fa-8930-eba9e5e61287","Type":"ContainerStarted","Data":"1c2f0e689e332a5e572e63049dee8655a77972b916fa15ef5e794fba5123486b"} Oct 30 05:53:02.798713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.798675 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:02.834467 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:02.834415 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podStartSLOduration=1.8344016459999999 podStartE2EDuration="1.834401646s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:02.832587263 +0000 UTC m=+1289.408274317" watchObservedRunningTime="2026-10-30 05:53:02.834401646 +0000 UTC m=+1289.410088699" Oct 30 05:53:03.804769 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:03.804740 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:04.757097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.757060 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:04.758102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.758077 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:53:04.907049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.907019 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc"] Oct 30 05:53:04.911978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.911957 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:04.949970 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.949936 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc"] Oct 30 05:53:04.958497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.958468 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf"] Oct 30 05:53:04.964752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.964729 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:04.994786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:04.994750 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf"] Oct 30 05:53:05.067013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.066939 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgfmw\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.066993 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067083 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067121 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067248 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067303 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsxlx\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067335 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067425 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067363 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067489 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067555 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067595 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067706 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067739 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067777 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067814 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.067868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067860 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.067971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067876 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.068030 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.067973 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.169594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169564 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgfmw\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.169729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169618 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.169729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169670 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.169729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169695 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.169851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169765 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.169851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169809 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsxlx\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.169975 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169859 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.169975 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169899 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170027 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.169975 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170092 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170129 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170170 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170240 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170268 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170269 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170341 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170385 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170398 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170500 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170505 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170575 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.170782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170632 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170659 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.170782 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.170726 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.171095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.171003 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.171163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.171132 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.171449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.171429 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.172591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.172572 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.172663 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.172593 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.173242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.173226 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.173391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.173372 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.195387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.195362 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.195619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.195603 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsxlx\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx\") pod \"maas-default-gateway-openshift-default-84587756dc-787sf\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.195784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.195764 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgfmw\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.196521 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.196503 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-96zxc\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.227200 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.227172 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:05.302216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.302188 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:05.626732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.626680 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:05.641389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.632200 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc"] Oct 30 05:53:05.641688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.641654 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:05.642636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.642607 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:05.662263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.662230 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf"] Oct 30 05:53:05.667468 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:05.667448 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d56599e_90f7_4183_a8d8_dff62f249f3b.slice/crio-3c80d0bf42a40cd517a037b1630ce96bf58869eade2072e8e628ec83824edee1 WatchSource:0}: Error finding container 3c80d0bf42a40cd517a037b1630ce96bf58869eade2072e8e628ec83824edee1: Status 404 returned error can't find the container with id 3c80d0bf42a40cd517a037b1630ce96bf58869eade2072e8e628ec83824edee1 Oct 30 05:53:05.670271 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.670241 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:05.670415 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.670311 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:05.670415 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.670345 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:05.787631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.781219 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" event={"ID":"c6f93246-910c-4e9d-9a41-386b3b8e00c7","Type":"ContainerStarted","Data":"ad76c4a42fd12d36b82e4e48cff476849379a29ecac9e775656fa5561f550910"} Oct 30 05:53:05.787631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.781283 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" event={"ID":"c6f93246-910c-4e9d-9a41-386b3b8e00c7","Type":"ContainerStarted","Data":"058294eecac3eaa77eff20eb8541166a996c28ef077ad61766e7980ee73908b8"} Oct 30 05:53:05.787631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.786364 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" event={"ID":"1d56599e-90f7-4183-a8d8-dff62f249f3b","Type":"ContainerStarted","Data":"b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692"} Oct 30 05:53:05.787631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.786410 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" event={"ID":"1d56599e-90f7-4183-a8d8-dff62f249f3b","Type":"ContainerStarted","Data":"3c80d0bf42a40cd517a037b1630ce96bf58869eade2072e8e628ec83824edee1"} Oct 30 05:53:05.851553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.849350 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podStartSLOduration=1.849330822 podStartE2EDuration="1.849330822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:05.844867157 +0000 UTC m=+1292.420554214" watchObservedRunningTime="2026-10-30 05:53:05.849330822 +0000 UTC m=+1292.425017878" Oct 30 05:53:05.896073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:05.896006 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podStartSLOduration=1.8959846649999998 podStartE2EDuration="1.895984665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:05.894148293 +0000 UTC m=+1292.469835346" watchObservedRunningTime="2026-10-30 05:53:05.895984665 +0000 UTC m=+1292.471671720" Oct 30 05:53:06.227622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:06.227544 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:06.229220 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:06.229193 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:06.229354 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:06.229244 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:06.303031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:06.303000 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:06.304334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:06.304307 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:06.304480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:06.304361 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:07.228013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.227982 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:07.228426 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.228034 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:07.307437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.307398 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:07.307747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.307716 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:07.824847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.824806 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl"] Oct 30 05:53:07.833184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.833143 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl"] Oct 30 05:53:07.837841 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.837754 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:07.869154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.868732 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47"] Oct 30 05:53:07.945920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.943221 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-b4ca14-dockercfg-fklvp\"" Oct 30 05:53:07.945920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.943444 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-b4ca14\"" Oct 30 05:53:07.946297 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.946270 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4n4t\" (UniqueName: \"kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t\") pod \"payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:07.946513 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.946476 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:07.951412 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.951385 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:07.961612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:07.961586 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47"] Oct 30 05:53:08.158597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.140961 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:08.158597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.145091 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4n4t\" (UniqueName: \"kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t\") pod \"payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:08.158597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.147365 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:08.244566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.240626 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:08.244566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.240685 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:08.249772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.249484 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htzxb\" (UniqueName: \"kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb\") pod \"payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:08.250790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.250057 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:08.265239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.265211 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4n4t\" (UniqueName: \"kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t\") pod \"payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:08.316168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.315624 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:08.316168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.315681 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:08.378609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.361742 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htzxb\" (UniqueName: \"kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb\") pod \"payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:08.379068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.379038 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:08.389928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.380870 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:08.407548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.404620 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htzxb\" (UniqueName: \"kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb\") pod \"payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:08.439647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.428950 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:08.720603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:08.719805 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:09.125316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.125271 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl"] Oct 30 05:53:09.219407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.214550 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47"] Oct 30 05:53:09.224454 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:09.224363 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29254e0c_3ea4_4b09_a9eb_4a199e28c85d.slice/crio-d95e8cd8d74b817d52555b3df7a8b8b3a8a54804e9081ad137e368962c02a717 WatchSource:0}: Error finding container d95e8cd8d74b817d52555b3df7a8b8b3a8a54804e9081ad137e368962c02a717: Status 404 returned error can't find the container with id d95e8cd8d74b817d52555b3df7a8b8b3a8a54804e9081ad137e368962c02a717 Oct 30 05:53:09.227853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.227828 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:09.227968 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.227870 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:09.312550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.310228 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:09.312550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.310284 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:09.342489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:09.342451 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:53:10.037636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.037584 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" event={"ID":"29254e0c-3ea4-4b09-a9eb-4a199e28c85d","Type":"ContainerStarted","Data":"9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198"} Oct 30 05:53:10.037985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.037941 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" event={"ID":"29254e0c-3ea4-4b09-a9eb-4a199e28c85d","Type":"ContainerStarted","Data":"d95e8cd8d74b817d52555b3df7a8b8b3a8a54804e9081ad137e368962c02a717"} Oct 30 05:53:10.038160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.038136 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:10.039956 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.039926 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" event={"ID":"950f39da-aad8-4ac4-93d2-98d47cf0789a","Type":"ContainerStarted","Data":"4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315"} Oct 30 05:53:10.040050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.039980 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" event={"ID":"950f39da-aad8-4ac4-93d2-98d47cf0789a","Type":"ContainerStarted","Data":"bd2dfe26cde2ee6f55541a34cfad336c990f238e2487e85d0a983e94d47aec1f"} Oct 30 05:53:10.040189 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.040169 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:10.115588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.115482 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" podStartSLOduration=3.115463221 podStartE2EDuration="3.115463221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:10.111513054 +0000 UTC m=+1296.687200112" watchObservedRunningTime="2026-10-30 05:53:10.115463221 +0000 UTC m=+1296.691150280" Oct 30 05:53:10.130351 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.130319 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc"] Oct 30 05:53:10.135435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.135415 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.152691 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.152667 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a1c17ebc\"" Oct 30 05:53:10.152976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.152927 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a1c17ebc-dockercfg-8hhnz\"" Oct 30 05:53:10.227716 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.216640 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc"] Oct 30 05:53:10.227716 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.224739 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" podStartSLOduration=3.224721792 podStartE2EDuration="3.224721792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:10.17106719 +0000 UTC m=+1296.746754244" watchObservedRunningTime="2026-10-30 05:53:10.224721792 +0000 UTC m=+1296.800408847" Oct 30 05:53:10.235692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.229832 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:10.235692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.229883 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:10.243159 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.243095 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw"] Oct 30 05:53:10.249273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.249248 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw"] Oct 30 05:53:10.249398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.249354 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.321235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.315913 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:10.321235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.315970 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:10.349585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.341102 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.349585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.341155 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nmlm\" (UniqueName: \"kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.349585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.341354 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2c782\" (UniqueName: \"kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.349585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.341403 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.463551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.463153 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2c782\" (UniqueName: \"kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.463551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.463229 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.463551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.463501 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.479555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.463546 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nmlm\" (UniqueName: \"kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.486559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.479915 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.519553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.504315 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.644605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.634256 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2c782\" (UniqueName: \"kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:10.670566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.668753 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nmlm\" (UniqueName: \"kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.789552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.785799 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:10.893006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:10.891798 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:11.264299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:11.247135 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:11.264299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:11.247193 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:11.511865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:11.490942 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:11.511865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:11.491023 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:12.068154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:12.047439 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc"] Oct 30 05:53:12.223552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:12.220798 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw"] Oct 30 05:53:12.261049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:12.240332 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:12.261049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:12.240385 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:12.329611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:12.328471 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:12.329611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:12.328545 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:13.101553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.101367 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq"] Oct 30 05:53:13.110876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.110843 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b37365e9-tnlgq"] Oct 30 05:53:13.121940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.121908 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" event={"ID":"99c45e58-fe46-4c9f-9dfa-53be9a553aff","Type":"ContainerStarted","Data":"9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16"} Oct 30 05:53:13.122084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.121971 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" event={"ID":"99c45e58-fe46-4c9f-9dfa-53be9a553aff","Type":"ContainerStarted","Data":"b042a159f68cdbaba79dba607063db88120bc45edd4406e52e22593b0bd326f2"} Oct 30 05:53:13.122362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.122340 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:13.123870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.123842 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" event={"ID":"6b832e8d-23d0-47e3-8f47-bae82bbc7e18","Type":"ContainerStarted","Data":"284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4"} Oct 30 05:53:13.123982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.123896 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" event={"ID":"6b832e8d-23d0-47e3-8f47-bae82bbc7e18","Type":"ContainerStarted","Data":"37ca1639cd426808c30484b059b1f4cc952955fe67e450c79e8df92e3b892146"} Oct 30 05:53:13.123982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.123927 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:13.228500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.228460 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:13.228704 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.228567 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:13.267011 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.266952 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" podStartSLOduration=3.2669350169999998 podStartE2EDuration="3.266935017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:13.196703263 +0000 UTC m=+1299.772390319" watchObservedRunningTime="2026-10-30 05:53:13.266935017 +0000 UTC m=+1299.842622073" Oct 30 05:53:13.268839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.268783 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" podStartSLOduration=3.268767288 podStartE2EDuration="3.268767288s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:13.257826013 +0000 UTC m=+1299.833513073" watchObservedRunningTime="2026-10-30 05:53:13.268767288 +0000 UTC m=+1299.844454343" Oct 30 05:53:13.302983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.302941 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:13.303160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.303010 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:13.572508 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.572472 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f79bf49-efb1-4ef2-bd25-a1d8eb504483" path="/var/lib/kubelet/pods/6f79bf49-efb1-4ef2-bd25-a1d8eb504483/volumes" Oct 30 05:53:13.979040 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.975923 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8"] Oct 30 05:53:13.982657 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:13.982630 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.020546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.020497 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-37bbcc-openshift-default-dockercfg-h7hzk\"" Oct 30 05:53:14.050605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.050575 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8"] Oct 30 05:53:14.075876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.075836 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.083779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.083740 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.083928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.083913 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.084025 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.084003 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.084158 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.084139 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.084218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.084202 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.084289 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.084267 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.084344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.084324 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.084390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.084359 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlkfp\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.185768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.185729 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.185793 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.185846 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.185881 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.185906 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlkfp\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.185989 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186050 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186118 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186233 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186318 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186585 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186711 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.186772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186710 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.187005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.186814 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.189215 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.189172 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.189756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.189733 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.223925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.223795 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlkfp\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.226956 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.226931 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.227780 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.227759 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-96zxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" start-of-body= Oct 30 05:53:14.227892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.227803 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.139:15021/healthz/ready\": dial tcp 10.134.0.139:15021: connect: connection refused" Oct 30 05:53:14.302598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.302556 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-787sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 30 05:53:14.302801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.302620 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 30 05:53:14.305104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.305074 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:14.678854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.678815 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8"] Oct 30 05:53:14.690401 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:14.689317 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bc5ae8_9e34_4b7e_a882_ccf1f28ecc4e.slice/crio-a9161c1e7e1aa08dd751e9d52112cbde6ca936b95a001c4d3c50f62afb80ebbf WatchSource:0}: Error finding container a9161c1e7e1aa08dd751e9d52112cbde6ca936b95a001c4d3c50f62afb80ebbf: Status 404 returned error can't find the container with id a9161c1e7e1aa08dd751e9d52112cbde6ca936b95a001c4d3c50f62afb80ebbf Oct 30 05:53:14.692175 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.692131 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:14.692301 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.692224 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:14.692301 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:14.692275 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:15.235547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.234778 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" event={"ID":"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e","Type":"ContainerStarted","Data":"c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a"} Oct 30 05:53:15.235547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.234852 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" event={"ID":"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e","Type":"ContainerStarted","Data":"a9161c1e7e1aa08dd751e9d52112cbde6ca936b95a001c4d3c50f62afb80ebbf"} Oct 30 05:53:15.383751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.383696 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:15.385988 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.385928 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podStartSLOduration=2.385909491 podStartE2EDuration="2.385909491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:15.381138532 +0000 UTC m=+1301.956825588" watchObservedRunningTime="2026-10-30 05:53:15.385909491 +0000 UTC m=+1301.961596546" Oct 30 05:53:15.389594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.389569 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:53:15.389826 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.389799 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:53:15.779380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.779290 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:15.780795 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.780286 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:15.786410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.784092 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 05:53:15.871913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.871877 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:15.872843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.872821 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:15.873993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:15.873976 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 05:53:16.099228 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:16.099130 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx"] Oct 30 05:53:16.105323 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:16.105254 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ae4dae-7nkpx"] Oct 30 05:53:16.310801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:16.310773 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:17.253601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:17.253568 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:17.254369 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:17.254348 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:53:17.533415 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:53:17.533330 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:53:17.569163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:17.569131 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="746fcd6a-7ea2-4caa-858b-b6c39b208d41" path="/var/lib/kubelet/pods/746fcd6a-7ea2-4caa-858b-b6c39b208d41/volumes" Oct 30 05:53:19.527862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.527831 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl"] Oct 30 05:53:19.533109 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.533089 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.552975 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.552947 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:53:19.559697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.559673 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.559913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.559895 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.560004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.559961 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nxrx\" (UniqueName: \"kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.568477 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.565637 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:53:19.572962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.572938 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl"] Oct 30 05:53:19.579204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.579183 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:53:19.579283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.579217 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:53:19.661934 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.661905 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nxrx\" (UniqueName: \"kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.662127 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.661996 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.662199 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.662148 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.662673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.662649 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.664522 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.664504 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.685971 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.685944 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nxrx\" (UniqueName: \"kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx\") pod \"maas-api-revoke-sub-dee2c484-zzqzl\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:19.848903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:19.848828 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:20.193539 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:20.193488 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb449b574_e46f_4d7c_9b76_8ff958a4f9fb.slice/crio-7011e32b8d339f737a678cc763f07e80bedfe2a7f16815607bc657ef2974aaa5 WatchSource:0}: Error finding container 7011e32b8d339f737a678cc763f07e80bedfe2a7f16815607bc657ef2974aaa5: Status 404 returned error can't find the container with id 7011e32b8d339f737a678cc763f07e80bedfe2a7f16815607bc657ef2974aaa5 Oct 30 05:53:20.196862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:20.196834 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl"] Oct 30 05:53:20.285877 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:20.281658 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" event={"ID":"b449b574-e46f-4d7c-9b76-8ff958a4f9fb","Type":"ContainerStarted","Data":"7011e32b8d339f737a678cc763f07e80bedfe2a7f16815607bc657ef2974aaa5"} Oct 30 05:53:21.066862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:21.066832 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 05:53:21.067935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:21.067908 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 05:53:21.294641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:21.294610 5565 generic.go:356] "Generic (PLEG): container finished" podID="b449b574-e46f-4d7c-9b76-8ff958a4f9fb" containerID="53ac9595bd1dc713bd980a0efae0045e63f5d26ffec690a2b295691be2d16bf6" exitCode=0 Oct 30 05:53:21.294787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:21.294697 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" event={"ID":"b449b574-e46f-4d7c-9b76-8ff958a4f9fb","Type":"ContainerDied","Data":"53ac9595bd1dc713bd980a0efae0045e63f5d26ffec690a2b295691be2d16bf6"} Oct 30 05:53:22.497075 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.497038 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:22.641162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.641064 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca\") pod \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " Oct 30 05:53:22.641162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.641147 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx\" (UniqueName: \"kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx\") pod \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " Oct 30 05:53:22.641419 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.641194 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca\") pod \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\" (UID: \"b449b574-e46f-4d7c-9b76-8ff958a4f9fb\") " Oct 30 05:53:22.642468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.641630 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca" pod "b449b574-e46f-4d7c-9b76-8ff958a4f9fb" (UID: "b449b574-e46f-4d7c-9b76-8ff958a4f9fb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:22.642468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.642118 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:53:22.644243 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.644077 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca" pod "b449b574-e46f-4d7c-9b76-8ff958a4f9fb" (UID: "b449b574-e46f-4d7c-9b76-8ff958a4f9fb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:22.645373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.645343 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx" pod "b449b574-e46f-4d7c-9b76-8ff958a4f9fb" (UID: "b449b574-e46f-4d7c-9b76-8ff958a4f9fb"). InnerVolumeSpecName "kube-api-access-8nxrx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:22.743625 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.743591 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:53:22.743625 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:22.743618 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nxrx\" (UniqueName: \"kubernetes.io/projected/b449b574-e46f-4d7c-9b76-8ff958a4f9fb-kube-api-access-8nxrx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:53:23.315251 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:23.315212 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" event={"ID":"b449b574-e46f-4d7c-9b76-8ff958a4f9fb","Type":"ContainerDied","Data":"7011e32b8d339f737a678cc763f07e80bedfe2a7f16815607bc657ef2974aaa5"} Oct 30 05:53:23.315435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:23.315264 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7011e32b8d339f737a678cc763f07e80bedfe2a7f16815607bc657ef2974aaa5" Oct 30 05:53:23.315435 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:23.315279 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl" Oct 30 05:53:24.138330 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:24.138302 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:53:24.138813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:24.138372 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:53:27.038032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:27.037999 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh"] Oct 30 05:53:27.050729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:27.050694 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a7f6db91-l5dqh"] Oct 30 05:53:27.055854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:27.055828 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2"] Oct 30 05:53:27.060009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:27.059989 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-32b3f894-f8kt2"] Oct 30 05:53:27.571193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:27.571162 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fb1c3d8-569f-4a14-89da-5cd9bbef3a79" path="/var/lib/kubelet/pods/0fb1c3d8-569f-4a14-89da-5cd9bbef3a79/volumes" Oct 30 05:53:27.571713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:27.571699 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99f98609-5305-4aad-86b0-aff342723ea4" path="/var/lib/kubelet/pods/99f98609-5305-4aad-86b0-aff342723ea4/volumes" Oct 30 05:53:30.780835 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.780794 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d"] Oct 30 05:53:30.781675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.781657 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b449b574-e46f-4d7c-9b76-8ff958a4f9fb" containerName="revoke-keys" Oct 30 05:53:30.781737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.781676 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b449b574-e46f-4d7c-9b76-8ff958a4f9fb" containerName="revoke-keys" Oct 30 05:53:30.781810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.781800 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b449b574-e46f-4d7c-9b76-8ff958a4f9fb" containerName="revoke-keys" Oct 30 05:53:30.787334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.787311 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.824484 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.824457 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d"] Oct 30 05:53:30.888123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.888094 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-0b481b-openshift-default-dockercfg-nrg7r\"" Oct 30 05:53:30.961469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961438 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.961644 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961591 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.961644 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961630 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.961750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961677 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.961800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961772 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.961909 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961886 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.961972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961930 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74j2s\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.962019 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.961985 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:30.962092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:30.962076 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063403 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063328 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063407 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063427 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063454 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063562 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063486 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063600 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063642 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74j2s\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063678 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.063791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063761 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.064006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063817 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.064006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.063930 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.064169 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.064137 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.064268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.064151 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.064567 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.064540 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.066253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.066232 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.066611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.066588 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.090052 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.090020 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.090498 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.090470 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74j2s\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.106917 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.106893 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:31.442907 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:31.442879 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67c3d184_a561_4c40_a950_6e93e7ae48de.slice/crio-d1ee25e210a9b6125325c01d48165318b6c5376c2abfb584fca2d60707a29a19 WatchSource:0}: Error finding container d1ee25e210a9b6125325c01d48165318b6c5376c2abfb584fca2d60707a29a19: Status 404 returned error can't find the container with id d1ee25e210a9b6125325c01d48165318b6c5376c2abfb584fca2d60707a29a19 Oct 30 05:53:31.444322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.444292 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d"] Oct 30 05:53:31.445719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.445686 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:31.445809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.445751 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:31.445809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:31.445787 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:32.040098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:32.040065 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr"] Oct 30 05:53:32.052660 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:32.052631 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-417adf23-fmpjr"] Oct 30 05:53:32.409520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:32.409429 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" event={"ID":"67c3d184-a561-4c40-a950-6e93e7ae48de","Type":"ContainerStarted","Data":"21ffeb60497120eb2906eff3a2fa46b4694f90cce39f1e19c6761d25eda0dfa1"} Oct 30 05:53:32.409520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:32.409514 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" event={"ID":"67c3d184-a561-4c40-a950-6e93e7ae48de","Type":"ContainerStarted","Data":"d1ee25e210a9b6125325c01d48165318b6c5376c2abfb584fca2d60707a29a19"} Oct 30 05:53:32.486661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:32.486591 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podStartSLOduration=2.486574676 podStartE2EDuration="2.486574676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:32.482167137 +0000 UTC m=+1319.057854202" watchObservedRunningTime="2026-10-30 05:53:32.486574676 +0000 UTC m=+1319.062261731" Oct 30 05:53:33.107579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.107543 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:33.109398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.109363 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 05:53:33.109724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.109686 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 05:53:33.588977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.588753 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf" path="/var/lib/kubelet/pods/bd7c9a2c-409e-4dcd-a9a9-23895bf4f0cf/volumes" Oct 30 05:53:33.798331 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.798296 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s"] Oct 30 05:53:33.804759 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.804735 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.829950 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.829917 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s"] Oct 30 05:53:33.836935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.836910 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-a14091-openshift-default-dockercfg-xdwgh\"" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.907506 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.907632 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.907794 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.907908 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.908072 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.908171 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.908265 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.908322 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:33.908439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:33.908421 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vddrh\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.009924 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.009891 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.009943 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.009972 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010106 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010009 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vddrh\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010105 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010138 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010267 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010220 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010287 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010419 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010425 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010478 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010672 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010615 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010724 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.010915 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.010754 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.012749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.012730 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.013073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.013051 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.044598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.039083 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.044598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.041029 5565 scope.go:122] "RemoveContainer" containerID="817f3b4a23c4baef40774109aa4d479f8a93baf33d9f620715f34f2db2b700f9" Oct 30 05:53:34.092548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.092145 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vddrh\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.099373 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.099338 5565 scope.go:122] "RemoveContainer" containerID="cd2e6b28a81a4d9b6f05cc8a22d4a17eff485f87c5682ee3364f51931b7c33ab" Oct 30 05:53:34.109939 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.109911 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 05:53:34.110393 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.110363 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 05:53:34.113101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.113074 5565 scope.go:122] "RemoveContainer" containerID="45f730e39b9906676fbaf38ad7d0ad05000818c0ab8e944a3d64a79cbf70e725" Oct 30 05:53:34.127922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.127887 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:34.128484 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.128394 5565 scope.go:122] "RemoveContainer" containerID="fca099f7856d11d6d46bc8caea0fa111d38700b82400ba17a7b13ee0b5849b05" Oct 30 05:53:34.584362 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:34.584333 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5490d6a_3257_42ba_83cf_dc71061f6835.slice/crio-1e747f10b2f173e801f9452f2ed77b8d9a6aac5aca43ec82062cd15faa6dfaa4 WatchSource:0}: Error finding container 1e747f10b2f173e801f9452f2ed77b8d9a6aac5aca43ec82062cd15faa6dfaa4: Status 404 returned error can't find the container with id 1e747f10b2f173e801f9452f2ed77b8d9a6aac5aca43ec82062cd15faa6dfaa4 Oct 30 05:53:34.585816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.585788 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s"] Oct 30 05:53:34.587148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.587122 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:34.587222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.587178 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:34.587222 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:34.587208 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:35.063157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.063091 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl"] Oct 30 05:53:35.117956 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.117912 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 05:53:35.118443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.118001 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 05:53:35.133593 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.133555 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7181d5eb-9fdjl"] Oct 30 05:53:35.444568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.444425 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" event={"ID":"f5490d6a-3257-42ba-83cf-dc71061f6835","Type":"ContainerStarted","Data":"f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8"} Oct 30 05:53:35.444568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.444538 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" event={"ID":"f5490d6a-3257-42ba-83cf-dc71061f6835","Type":"ContainerStarted","Data":"1e747f10b2f173e801f9452f2ed77b8d9a6aac5aca43ec82062cd15faa6dfaa4"} Oct 30 05:53:35.627623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:35.627590 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5dafc59-5f16-4d11-9bb4-266aef5ce9a3" path="/var/lib/kubelet/pods/f5dafc59-5f16-4d11-9bb4-266aef5ce9a3/volumes" Oct 30 05:53:36.110549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:36.109939 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.110549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:36.109998 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 05:53:36.129350 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:36.129314 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:36.130755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:36.130728 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:36.130873 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:36.130779 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:37.119719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.096947 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podStartSLOduration=4.096925634 podStartE2EDuration="4.096925634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:35.508589021 +0000 UTC m=+1322.084276078" watchObservedRunningTime="2026-10-30 05:53:37.096925634 +0000 UTC m=+1323.672612689" Oct 30 05:53:37.119719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.116444 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.119719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.119335 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" Oct 30 05:53:37.128571 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.128505 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz"] Oct 30 05:53:37.129218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.129027 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:37.129218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.129074 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:37.134308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.134286 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-61fed433-t5tsz"] Oct 30 05:53:37.623916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:37.623870 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9ec65e1-f683-4166-824e-ccfa6bcce7c4" path="/var/lib/kubelet/pods/c9ec65e1-f683-4166-824e-ccfa6bcce7c4/volumes" Oct 30 05:53:38.114833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:38.114808 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:38.115345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:38.115320 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:38.116276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:38.116254 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 05:53:38.129866 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:38.129840 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:38.130305 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:38.129896 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:39.129087 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:39.129056 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:39.129279 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:39.129111 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:40.129384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:40.129347 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:40.129819 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:40.129407 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:41.129437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:41.129397 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:41.130174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:41.130139 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:42.137574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:42.137521 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" start-of-body= Oct 30 05:53:42.147803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:42.147761 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": dial tcp 10.134.0.148:15021: connect: connection refused" Oct 30 05:53:42.979669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:42.979620 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.013290 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.013247 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.015787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.015757 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.036891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.036851 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.040630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.037109 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.084641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.083401 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.084641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.083573 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.138516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.130478 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.138516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.130544 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.225572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.218278 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.355678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.355640 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs"] Oct 30 05:53:43.384455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.384414 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:43.602837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.602798 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng"] Oct 30 05:53:43.648620 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:53:43.635332 5565 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-sub-7b996d00-8zvzs\" is forbidden: User \"system:node:ip-10-0-129-175.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-129-175.ec2.internal' and this object" podUID="421b838d-7dbd-4ca9-9046-25bfd4abe129" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs" Oct 30 05:53:43.948253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:43.948148 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b996d00-8zvzs"] Oct 30 05:53:44.017553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.017039 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 05:53:44.027550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.018175 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:53:44.031549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.028738 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d64f561f-bp8ng"] Oct 30 05:53:44.031549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.028837 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.031549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.029188 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:53:44.041146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.041111 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.041749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.041731 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:53:44.069079 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.069043 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:53:44.085452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.085415 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.086244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.086222 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:53:44.087111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.087083 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.087541 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.087506 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:53:44.147655 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.147604 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:53:44.148160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.148051 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:53:44.170979 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.170954 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:53:44.188462 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.188436 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:53:44.226736 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.223253 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 05:53:44.237504 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.233544 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:53:44.255560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.243512 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:53:44.255560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.243582 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:53:44.275611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.263293 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:53:44.310659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.310626 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:53:44.385292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.385256 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:53:44.410145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.409973 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:53:44.483891 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.483809 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:44.484431 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.484407 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:44.485205 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:44.485186 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 05:53:45.569506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:45.569475 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="421b838d-7dbd-4ca9-9046-25bfd4abe129" path="/var/lib/kubelet/pods/421b838d-7dbd-4ca9-9046-25bfd4abe129/volumes" Oct 30 05:53:45.569994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:45.569979 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5703bb0b-42cb-4e32-8045-9103f17cf1ce" path="/var/lib/kubelet/pods/5703bb0b-42cb-4e32-8045-9103f17cf1ce/volumes" Oct 30 05:53:47.404514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.404479 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm"] Oct 30 05:53:47.412314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.412289 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.431721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.431690 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.432312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.432036 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4w8m\" (UniqueName: \"kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.432312 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.432292 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.432639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.432337 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.440908 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.440877 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm"] Oct 30 05:53:47.446355 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.446328 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 05:53:47.446624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.446604 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-37bbcc-serving-cert\"" Oct 30 05:53:47.447101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.447082 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 05:53:47.458937 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.458913 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wbd8\"" Oct 30 05:53:47.459073 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.458965 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 05:53:47.533495 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.533460 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.533727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.533515 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.533820 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.533787 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.534149 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.533874 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4w8m\" (UniqueName: \"kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.535665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.535189 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.537771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.537747 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.537941 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.537917 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.546244 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:53:47.546210 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:53:47.562374 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.561965 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4w8m\" (UniqueName: \"kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m\") pod \"maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:47.731839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:47.731736 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:48.062706 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:53:48.062674 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod599547ed_d597_42b6_a585_bec48f04059e.slice/crio-1ad0aeb7222417206541c466788af5271c44f32d31b689dc9bd6e83bc4647d05 WatchSource:0}: Error finding container 1ad0aeb7222417206541c466788af5271c44f32d31b689dc9bd6e83bc4647d05: Status 404 returned error can't find the container with id 1ad0aeb7222417206541c466788af5271c44f32d31b689dc9bd6e83bc4647d05 Oct 30 05:53:48.063745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.063706 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm"] Oct 30 05:53:48.880612 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.879770 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s"] Oct 30 05:53:48.890985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.890951 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:48.924055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.924014 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm"] Oct 30 05:53:48.924592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.924256 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:53:48.924592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.924265 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:53:48.936129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.932147 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s"] Oct 30 05:53:48.936129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.932283 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.947790 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.946769 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm"] Oct 30 05:53:48.970583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.970523 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.970986 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.970955 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.971418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.971368 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgccz\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.971781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.971760 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.972054 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.972034 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.972384 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.972362 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.973958 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.973919 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.974234 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.974206 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:48.974416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.974387 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr2pq\" (UniqueName: \"kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:48.974636 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.974617 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:48.974872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.974851 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:48.975088 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.975054 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.002558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:48.999199 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" event={"ID":"599547ed-d597-42b6-a585-bec48f04059e","Type":"ContainerStarted","Data":"1ad0aeb7222417206541c466788af5271c44f32d31b689dc9bd6e83bc4647d05"} Oct 30 05:53:49.076589 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076513 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.076781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076642 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.076781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076757 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgccz\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.076925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076821 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.076925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076853 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.076925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076919 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.077069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.076997 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.077069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.077058 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.077183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.077083 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr2pq\" (UniqueName: \"kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.077183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.077106 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.077183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.077140 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.077322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.077192 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.077881 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.077855 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.078089 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.078063 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.078259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.078240 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.079347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.079325 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.079460 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.079322 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.079869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.079839 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.080145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.080125 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.082339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.082281 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.082339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.082298 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.110292 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.110257 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr2pq\" (UniqueName: \"kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq\") pod \"maas-api-revoke-sub-15e2832d-dcq8s\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.110960 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.110910 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgccz\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.111499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.111472 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token\") pod \"e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.229576 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.229458 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:49.267265 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.267229 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:49.594128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.594098 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s"] Oct 30 05:53:49.616322 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.616291 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm"] Oct 30 05:53:49.625395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.625354 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:49.625566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.625440 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:49.625566 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:49.625482 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:53:50.014640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.014598 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" event={"ID":"c935779e-20ad-42e6-982f-74ca32d9ded2","Type":"ContainerStarted","Data":"85844e598f1be93ea739ecfecee2cb22702db5204f43c14c09fe2d83e43e665f"} Oct 30 05:53:50.016976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.016947 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" event={"ID":"1ddb6c6b-95c6-4022-a4b6-0228a827cf85","Type":"ContainerStarted","Data":"12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e"} Oct 30 05:53:50.017131 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.017001 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" event={"ID":"1ddb6c6b-95c6-4022-a4b6-0228a827cf85","Type":"ContainerStarted","Data":"4795d8aa7802cacf14041659cf512ff860e5e47e5dbeffc5363c886bc9e50d9f"} Oct 30 05:53:50.068198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.068135 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podStartSLOduration=2.06811475 podStartE2EDuration="2.06811475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:50.065743108 +0000 UTC m=+1336.641430161" watchObservedRunningTime="2026-10-30 05:53:50.06811475 +0000 UTC m=+1336.643801806" Oct 30 05:53:50.267789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.267761 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:50.269450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.269418 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:50.269617 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:50.269475 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:51.033845 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:51.033811 5565 generic.go:356] "Generic (PLEG): container finished" podID="c935779e-20ad-42e6-982f-74ca32d9ded2" containerID="b922ae2d719c71b262c2ba8487274857e55f904ce2cc0b27f96e764a9b1a9f4f" exitCode=0 Oct 30 05:53:51.034319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:51.033895 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" event={"ID":"c935779e-20ad-42e6-982f-74ca32d9ded2","Type":"ContainerDied","Data":"b922ae2d719c71b262c2ba8487274857e55f904ce2cc0b27f96e764a9b1a9f4f"} Oct 30 05:53:51.278412 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:51.278370 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:51.278684 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:51.278434 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:52.048083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.048047 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" event={"ID":"599547ed-d597-42b6-a585-bec48f04059e","Type":"ContainerStarted","Data":"98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a"} Oct 30 05:53:52.102565 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.099573 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" podStartSLOduration=1.640049906 podStartE2EDuration="5.099551617s" totalImagesPullingTime="3.459501711s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:53:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:53:52.098947033 +0000 UTC m=+1338.674634087" watchObservedRunningTime="2026-10-30 05:53:52.099551617 +0000 UTC m=+1338.675238672" Oct 30 05:53:52.292551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.282842 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:52.292551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.282917 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:52.381083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.381051 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:52.450697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.450606 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq\" (UniqueName: \"kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq\") pod \"c935779e-20ad-42e6-982f-74ca32d9ded2\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " Oct 30 05:53:52.450870 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.450806 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca\") pod \"c935779e-20ad-42e6-982f-74ca32d9ded2\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " Oct 30 05:53:52.450937 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.450908 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca\") pod \"c935779e-20ad-42e6-982f-74ca32d9ded2\" (UID: \"c935779e-20ad-42e6-982f-74ca32d9ded2\") " Oct 30 05:53:52.451173 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.451143 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca" pod "c935779e-20ad-42e6-982f-74ca32d9ded2" (UID: "c935779e-20ad-42e6-982f-74ca32d9ded2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:53:52.451784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.451760 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:53:52.483816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.480612 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca" pod "c935779e-20ad-42e6-982f-74ca32d9ded2" (UID: "c935779e-20ad-42e6-982f-74ca32d9ded2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:53:52.483816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.481722 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq" pod "c935779e-20ad-42e6-982f-74ca32d9ded2" (UID: "c935779e-20ad-42e6-982f-74ca32d9ded2"). InnerVolumeSpecName "kube-api-access-gr2pq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:53:52.578948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.578866 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gr2pq\" (UniqueName: \"kubernetes.io/projected/c935779e-20ad-42e6-982f-74ca32d9ded2-kube-api-access-gr2pq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:53:52.578948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:52.578895 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c935779e-20ad-42e6-982f-74ca32d9ded2-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:53:53.079833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:53.079672 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" Oct 30 05:53:53.079833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:53.079697 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s" event={"ID":"c935779e-20ad-42e6-982f-74ca32d9ded2","Type":"ContainerDied","Data":"85844e598f1be93ea739ecfecee2cb22702db5204f43c14c09fe2d83e43e665f"} Oct 30 05:53:53.079833 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:53.079752 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="85844e598f1be93ea739ecfecee2cb22702db5204f43c14c09fe2d83e43e665f" Oct 30 05:53:53.080457 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:53.080035 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:53.273548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:53.271885 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:53.273548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:53.271945 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:54.277603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:54.276705 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:54.277603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:54.276769 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:55.277933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:55.277884 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:55.278446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:55.277951 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:56.267764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:56.267728 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 05:53:56.267965 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:56.267785 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 05:53:57.413421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:57.413387 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:57.414485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:57.414458 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:57.415471 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:57.415406 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:53:58.038584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:58.038547 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4"] Oct 30 05:53:58.051912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:58.051884 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9ea81c2-lpng4"] Oct 30 05:53:58.056860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:58.056833 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws"] Oct 30 05:53:58.062444 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:58.062414 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0cf7d6b8-zr6ws"] Oct 30 05:53:59.094809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:59.094777 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 05:53:59.569885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:59.569853 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ec3ab3e-29c3-4771-a4f8-ad5467873cd8" path="/var/lib/kubelet/pods/6ec3ab3e-29c3-4771-a4f8-ad5467873cd8/volumes" Oct 30 05:53:59.570349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:53:59.570334 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93d6d5c3-8cac-4e2d-aab2-94857cae53c1" path="/var/lib/kubelet/pods/93d6d5c3-8cac-4e2d-aab2-94857cae53c1/volumes" Oct 30 05:54:10.038480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:10.038440 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc"] Oct 30 05:54:10.050702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:10.050671 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc6b5c6a-wzdkc"] Oct 30 05:54:11.572539 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:11.572500 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93e3e226-ff88-4e30-a0d5-2e1d3968092a" path="/var/lib/kubelet/pods/93e3e226-ff88-4e30-a0d5-2e1d3968092a/volumes" Oct 30 05:54:15.961845 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:15.961759 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z"] Oct 30 05:54:15.962813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:15.962788 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c935779e-20ad-42e6-982f-74ca32d9ded2" containerName="revoke-keys" Oct 30 05:54:15.962813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:15.962811 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c935779e-20ad-42e6-982f-74ca32d9ded2" containerName="revoke-keys" Oct 30 05:54:15.963016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:15.962999 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c935779e-20ad-42e6-982f-74ca32d9ded2" containerName="revoke-keys" Oct 30 05:54:15.972061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:15.972038 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:15.993167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:15.993142 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z"] Oct 30 05:54:16.014609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.014575 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.014746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.014684 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.014746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.014718 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.014867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.014753 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.015049 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.014858 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.015810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.015786 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.016015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.015995 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrbgd\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.018710 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.018688 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.018917 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.018891 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.120800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120771 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.120962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120819 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.120962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120866 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.120962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120899 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.120962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120924 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.120962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120951 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.120988 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121059 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrbgd\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121122 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121328 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121444 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121513 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121693 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121669 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.121753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.121707 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.123498 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.123476 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.123585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.123478 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.208547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.208092 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.213441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.212240 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrbgd\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd\") pod \"e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.292137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.291865 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:16.683230 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.683206 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z"] Oct 30 05:54:16.685826 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:54:16.685797 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c0d1815_03a6_4485_86d0_c59b51432a58.slice/crio-9cc6706b0f2dd96b136bda935fbbad6fdccbc429244edfdd7efe07078224dcb2 WatchSource:0}: Error finding container 9cc6706b0f2dd96b136bda935fbbad6fdccbc429244edfdd7efe07078224dcb2: Status 404 returned error can't find the container with id 9cc6706b0f2dd96b136bda935fbbad6fdccbc429244edfdd7efe07078224dcb2 Oct 30 05:54:16.688974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.688928 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:16.689077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.689039 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:16.689121 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:16.689086 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:17.107594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:17.107420 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:54:17.356851 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:17.356812 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" event={"ID":"2c0d1815-03a6-4485-86d0-c59b51432a58","Type":"ContainerStarted","Data":"4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0"} Oct 30 05:54:17.357029 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:17.356881 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" event={"ID":"2c0d1815-03a6-4485-86d0-c59b51432a58","Type":"ContainerStarted","Data":"9cc6706b0f2dd96b136bda935fbbad6fdccbc429244edfdd7efe07078224dcb2"} Oct 30 05:54:17.417791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:17.417666 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podStartSLOduration=2.417646607 podStartE2EDuration="2.417646607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:17.412325998 +0000 UTC m=+1363.988013053" watchObservedRunningTime="2026-10-30 05:54:17.417646607 +0000 UTC m=+1363.993333686" Oct 30 05:54:17.542710 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:54:17.542671 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:54:18.386183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:18.386149 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:18.386810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:18.386659 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:18.386810 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:18.386709 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:19.295885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:19.295845 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:19.296115 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:19.295910 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:20.354815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:20.293061 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:20.354815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:20.293122 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:20.484920 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:20.483737 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv"] Oct 30 05:54:20.627605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:20.624764 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa0cb2d9-6r7wv"] Oct 30 05:54:20.981772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:20.981661 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.016880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.016836 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.099705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.099619 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:54:21.106134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.099951 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.122901 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.101044 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.241740 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.241637 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.285559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.285086 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.285559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.285218 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:54:21.342556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.290597 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.342556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.295213 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:21.342556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.295264 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:21.342556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.298621 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.390575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.386731 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:54:21.426686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.426630 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.628751 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.628704 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 05:54:21.961930 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.961844 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15569a1c-0527-479d-9333-59be9455ff40" path="/var/lib/kubelet/pods/15569a1c-0527-479d-9333-59be9455ff40/volumes" Oct 30 05:54:21.986680 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.986608 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:21.987102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:21.987076 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:54:22.022699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.022657 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.024084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.024062 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:54:22.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.241687 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.242101 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.242736 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:54:22.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.242795 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:54:22.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.268640 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:54:22.269616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.269079 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:54:22.288584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.275191 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.289913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.288920 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.289913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.289025 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.289913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.289522 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.290650 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.290623 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:54:22.311647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.303915 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:54:22.311647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.303975 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:54:22.311647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.306172 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.311647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.306452 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.319549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.313994 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:22.319549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.314040 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:22.319549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.314144 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:54:22.319549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.314193 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:54:22.491451 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.465622 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:22.503552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.492143 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:54:22.542617 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.519107 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 30 05:54:22.568786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:22.560654 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:54:23.014695 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.014645 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:23.053554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.023063 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:23.297105 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.245916 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:23.297105 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.247571 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:23.301021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.300983 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:54:23.317597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.305352 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:54:23.340575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.318846 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:54:23.340575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.318916 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 05:54:23.340575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.337417 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:23.340575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.318960 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:54:23.359449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.359176 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:23.359449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.359226 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:23.480438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.479639 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:54:23.868623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:23.739135 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.057555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.051055 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:54:24.260392 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.260346 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:54:24.293583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.271225 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.321906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.321861 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.357554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.341644 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:54:24.357554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.343109 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/health\": context deadline exceeded" Oct 30 05:54:24.357554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.343394 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:54:24.357554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.349052 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 05:54:24.357554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.349827 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:54:24.459682 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.459628 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:24.460041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.460005 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:24.510608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.484942 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.809609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.725857 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.859909 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.843639 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:24.875661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:24.843682 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.176396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.176285 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.207555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.178395 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:54:25.226276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.217905 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": context deadline exceeded" Oct 30 05:54:25.260374 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:54:25.259325 5565 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.63s" Oct 30 05:54:25.261628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.261590 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.293383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.293349 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:25.293671 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.293641 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:25.314549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.308626 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.347714 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.347665 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 05:54:25.416741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.416680 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.420572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.417418 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:54:25.429951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.429848 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": context deadline exceeded" Oct 30 05:54:25.452081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.452039 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.505554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.468650 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:54:25.734709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.734606 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:25.912668 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:25.912621 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.180560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.179297 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.229733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.229685 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.262681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.262631 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.264363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.264316 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.265445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.264583 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.333879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.333831 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.368549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.344563 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:26.368549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.344626 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:26.368549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.354476 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.417662 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.417612 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.453623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.440784 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.532575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.504726 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.577557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.531930 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:54:26.735753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.735642 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": context deadline exceeded" Oct 30 05:54:26.879858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.879806 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:26.954553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:26.939670 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.181406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.181361 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.233548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.233232 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.279607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.263341 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.279607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.265600 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.283678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.283641 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 05:54:27.330974 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.330925 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" start-of-body= Oct 30 05:54:27.331310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.331279 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.152:15021/healthz/ready\": dial tcp 10.134.0.152:15021: connect: connection refused" Oct 30 05:54:27.337779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.337744 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.376845 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.376788 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.431225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.431182 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.449676 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.449583 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.543977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.543325 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.604784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.604113 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.621729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.621337 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.749044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.748958 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 05:54:27.750719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.750696 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 05:54:27.752673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.752651 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl"] Oct 30 05:54:27.757628 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.757603 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" Oct 30 05:54:27.804114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.804080 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2c4e33c-tnzbl"] Oct 30 05:54:27.804391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.804371 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8"] Oct 30 05:54:27.804577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.804562 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 05:54:27.804699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.804687 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67624b-6ndt8"] Oct 30 05:54:27.804852 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.804841 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 05:54:27.821708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.819811 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 05:54:27.821708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.820681 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 05:54:27.849215 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.847683 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:27.853640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:27.850124 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 05:54:28.049178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:28.049150 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 05:54:28.140548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:28.140367 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 05:54:28.209670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:28.209545 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 05:54:28.321347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:28.321264 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:28.322423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:28.322401 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:28.324234 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:28.324181 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 05:54:29.488933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.488895 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl"] Oct 30 05:54:29.495179 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.495148 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.499255 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.499225 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb"] Oct 30 05:54:29.504079 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.504057 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.524976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.524944 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl"] Oct 30 05:54:29.542608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.529692 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-e94bd8\"" Oct 30 05:54:29.542608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.529728 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-e94bd8-dockercfg-kxhtn\"" Oct 30 05:54:29.542608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.533820 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.542608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.533871 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.542608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.534108 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgxn\" (UniqueName: \"kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn\") pod \"payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.542608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.534143 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xflh8\" (UniqueName: \"kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8\") pod \"payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.581329 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.581293 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3260ead5-1f24-4f40-a728-b152c725fb36" path="/var/lib/kubelet/pods/3260ead5-1f24-4f40-a728-b152c725fb36/volumes" Oct 30 05:54:29.583880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.582087 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29cee4e-2980-4266-997b-f017c448100b" path="/var/lib/kubelet/pods/a29cee4e-2980-4266-997b-f017c448100b/volumes" Oct 30 05:54:29.583880 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.582699 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb"] Oct 30 05:54:29.585134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.584227 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf"] Oct 30 05:54:29.609772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.609742 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.645719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.645689 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.646041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.646016 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgxn\" (UniqueName: \"kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn\") pod \"payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.646184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.646168 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xflh8\" (UniqueName: \"kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8\") pod \"payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.646325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.646310 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.646557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.646516 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.646787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.646770 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.646998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.646981 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.647125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.647110 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.647252 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.647237 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.647428 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.647414 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqq9\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.647623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.647606 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.647756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.647742 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.647905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.647890 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.650068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.650045 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume\") pod \"payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.651134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.651092 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.763718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.763682 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xflh8\" (UniqueName: \"kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8\") pod \"payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.769909 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.769872 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770091 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770017 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770166 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770109 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqq9\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770269 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770249 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770337 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770308 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770397 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770354 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770407 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770682 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770656 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.770809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.770767 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.771352 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.771324 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.772339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.771914 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.772797 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.772777 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf"] Oct 30 05:54:29.773272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.773249 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.773569 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.773499 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgxn\" (UniqueName: \"kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn\") pod \"payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:29.773733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.773558 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.773969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.773846 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.774783 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.774763 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.774783 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.774773 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.804099 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.804070 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqq9\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.806670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.806615 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token\") pod \"e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.807677 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.807446 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:29.828993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.828908 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:29.862348 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:29.861859 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:30.194584 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:54:30.194548 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f54885_68d7_439b_b786_d314d7af6eb0.slice/crio-a39cb8066f4cf2884b25898d31383abd08b56deb75b5b1a174c0ab1cdaeaf7a1 WatchSource:0}: Error finding container a39cb8066f4cf2884b25898d31383abd08b56deb75b5b1a174c0ab1cdaeaf7a1: Status 404 returned error can't find the container with id a39cb8066f4cf2884b25898d31383abd08b56deb75b5b1a174c0ab1cdaeaf7a1 Oct 30 05:54:30.200160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.198522 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:30.200160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.198623 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:30.200160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.198663 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:30.203442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.203418 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf"] Oct 30 05:54:30.243549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.243398 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" event={"ID":"40f54885-68d7-439b-b786-d314d7af6eb0","Type":"ContainerStarted","Data":"a39cb8066f4cf2884b25898d31383abd08b56deb75b5b1a174c0ab1cdaeaf7a1"} Oct 30 05:54:30.415042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.415014 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl"] Oct 30 05:54:30.416472 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:54:30.416375 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602f29b3_b904_4951_930a_b7b541ed6c9f.slice/crio-fd805a57d9efdd2f8adbab4085693ed427fcc6b241ab05386dc9363e3eb39b97 WatchSource:0}: Error finding container fd805a57d9efdd2f8adbab4085693ed427fcc6b241ab05386dc9363e3eb39b97: Status 404 returned error can't find the container with id fd805a57d9efdd2f8adbab4085693ed427fcc6b241ab05386dc9363e3eb39b97 Oct 30 05:54:30.428667 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:54:30.428642 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bf047a4_573a_4f50_837f_2af39aeceb58.slice/crio-bded2aa16097d413cb943851fc1ed45de03d6dfb4c4ab30875448453ded6b141 WatchSource:0}: Error finding container bded2aa16097d413cb943851fc1ed45de03d6dfb4c4ab30875448453ded6b141: Status 404 returned error can't find the container with id bded2aa16097d413cb943851fc1ed45de03d6dfb4c4ab30875448453ded6b141 Oct 30 05:54:30.431465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:30.431395 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb"] Oct 30 05:54:31.260092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.260055 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" event={"ID":"4bf047a4-573a-4f50-837f-2af39aeceb58","Type":"ContainerStarted","Data":"7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7"} Oct 30 05:54:31.260610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.260121 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" event={"ID":"4bf047a4-573a-4f50-837f-2af39aeceb58","Type":"ContainerStarted","Data":"bded2aa16097d413cb943851fc1ed45de03d6dfb4c4ab30875448453ded6b141"} Oct 30 05:54:31.261595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.260443 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:31.262640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.262614 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" event={"ID":"602f29b3-b904-4951-930a-b7b541ed6c9f","Type":"ContainerStarted","Data":"2f0155ea57daf7df990657d1ee17196f2d46465254f44da2889e4e1304a9ac5a"} Oct 30 05:54:31.262847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.262829 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" event={"ID":"602f29b3-b904-4951-930a-b7b541ed6c9f","Type":"ContainerStarted","Data":"fd805a57d9efdd2f8adbab4085693ed427fcc6b241ab05386dc9363e3eb39b97"} Oct 30 05:54:31.263014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.263002 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:31.266147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.266117 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" event={"ID":"40f54885-68d7-439b-b786-d314d7af6eb0","Type":"ContainerStarted","Data":"77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03"} Oct 30 05:54:31.304280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.304223 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" podStartSLOduration=2.304209241 podStartE2EDuration="2.304209241s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:31.302906255 +0000 UTC m=+1377.878593302" watchObservedRunningTime="2026-10-30 05:54:31.304209241 +0000 UTC m=+1377.879896295" Oct 30 05:54:31.362063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.362009 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" podStartSLOduration=2.361995692 podStartE2EDuration="2.361995692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:31.356452085 +0000 UTC m=+1377.932139154" watchObservedRunningTime="2026-10-30 05:54:31.361995692 +0000 UTC m=+1377.937682746" Oct 30 05:54:31.428238 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.426326 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podStartSLOduration=2.426311063 podStartE2EDuration="2.426311063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:31.417981327 +0000 UTC m=+1377.993668382" watchObservedRunningTime="2026-10-30 05:54:31.426311063 +0000 UTC m=+1378.001998117" Oct 30 05:54:31.807674 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.807633 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:31.809315 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.809278 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:54:31.809475 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:31.809343 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 05:54:32.032778 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:32.032729 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr"] Oct 30 05:54:32.040216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:32.040175 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-88a7c916-7vzxr"] Oct 30 05:54:32.808561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:32.808501 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:54:32.808978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:32.808594 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 05:54:33.300689 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:33.300370 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:33.300689 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:33.300427 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:33.595005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:33.594932 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf1001f-23fd-4f6d-ba7d-e5b80d3198dd" path="/var/lib/kubelet/pods/daf1001f-23fd-4f6d-ba7d-e5b80d3198dd/volumes" Oct 30 05:54:33.809608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:33.808976 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:54:33.809608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:33.809037 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 05:54:34.352259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.352222 5565 scope.go:122] "RemoveContainer" containerID="c5345a07cfd17e9fee41cce0cbddeb643c511dc62bd7b342e9c1003fc911e7d4" Oct 30 05:54:34.391720 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.391694 5565 scope.go:122] "RemoveContainer" containerID="7eb52a08fbe884f5b4513ff3d7d7c251126096d63f8a5e64fe4fa856d3b9b2b0" Oct 30 05:54:34.402660 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.402642 5565 scope.go:122] "RemoveContainer" containerID="53f279b1b80591e992e98a8347bbf0aff699edf800ee73a61c8868fffbc850b1" Oct 30 05:54:34.413183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.413165 5565 scope.go:122] "RemoveContainer" containerID="cacc40b5a94bb554a61951f8eb84b3fe54ee47fe8ce3b4dfc7ffb298d2d5fdfc" Oct 30 05:54:34.424217 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.424120 5565 scope.go:122] "RemoveContainer" containerID="76521c544530547bf25237536cd1c8e33178ace3a08cefeadf834fabcaa05e8f" Oct 30 05:54:34.436048 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.436021 5565 scope.go:122] "RemoveContainer" containerID="6b3507cecfaa7c8f481abae0599f79409bf81b6b203328f765039c27135f6faf" Oct 30 05:54:34.480014 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.479984 5565 scope.go:122] "RemoveContainer" containerID="fd3b5580e1195d8dff3fbe6c57cb82ee9d0af3461db197fbf3e07a0ac78544e8" Oct 30 05:54:34.500985 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.500960 5565 scope.go:122] "RemoveContainer" containerID="d42ae00ec4024ff5dac2c3aa101cf2715b3bf287a854988d7c1af830019daffb" Oct 30 05:54:34.514147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.514121 5565 scope.go:122] "RemoveContainer" containerID="22f212fc610160adc9d3f06ad701154ccc84c5edf15bd0cb2217b7586f8905fc" Oct 30 05:54:34.808359 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.808323 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:54:34.808539 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:34.808382 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 05:54:35.808332 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:35.808299 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:54:35.808963 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:35.808882 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 05:54:35.815681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:35.815650 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:54:35.815787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:35.815698 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:54:36.395480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.395449 5565 generic.go:356] "Generic (PLEG): container finished" podID="2d65898f-a620-49fa-8930-eba9e5e61287" containerID="35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927" exitCode=0 Oct 30 05:54:36.395753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.395521 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" event={"ID":"2d65898f-a620-49fa-8930-eba9e5e61287","Type":"ContainerDied","Data":"35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927"} Oct 30 05:54:36.396294 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.396274 5565 scope.go:122] "RemoveContainer" containerID="35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927" Oct 30 05:54:36.398376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.398344 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:36.398574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.398432 5565 generic.go:356] "Generic (PLEG): container finished" podID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerID="c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a" exitCode=0 Oct 30 05:54:36.398746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.398451 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" event={"ID":"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e","Type":"ContainerDied","Data":"c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a"} Oct 30 05:54:36.398900 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.398558 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:36.398976 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.398954 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:36.399355 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.399337 5565 scope.go:122] "RemoveContainer" containerID="c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a" Oct 30 05:54:36.401933 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.401910 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:36.402080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.402066 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:36.402191 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.402178 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:54:36.790607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.790570 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5"] Oct 30 05:54:36.800310 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.800282 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5"] Oct 30 05:54:36.800643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.800621 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.804935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.804908 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlgx8\" (UniqueName: \"kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.805217 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.805195 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.805422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.805399 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.808939 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.808070 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" start-of-body= Oct 30 05:54:36.808939 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.808145 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.155:15021/healthz/ready\": dial tcp 10.134.0.155:15021: connect: connection refused" Oct 30 05:54:36.822339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.822311 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:54:36.822447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.822433 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:54:36.907260 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.907207 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlgx8\" (UniqueName: \"kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.907464 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.907377 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.907464 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.907408 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.908416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.908391 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.909883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.909859 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:36.933128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:36.933099 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlgx8\" (UniqueName: \"kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8\") pod \"maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:37.127975 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.127896 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:37.444140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.444107 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" event={"ID":"2d65898f-a620-49fa-8930-eba9e5e61287","Type":"ContainerStarted","Data":"af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456"} Oct 30 05:54:37.447140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.447109 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" event={"ID":"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e","Type":"ContainerStarted","Data":"9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f"} Oct 30 05:54:37.620688 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:54:37.620650 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5fb421_9f59_458f_8516_32ce09653507.slice/crio-c162abf356a356e23c659137dd488d5c632488a160569c360c6a31f3a5b4687d WatchSource:0}: Error finding container c162abf356a356e23c659137dd488d5c632488a160569c360c6a31f3a5b4687d: Status 404 returned error can't find the container with id c162abf356a356e23c659137dd488d5c632488a160569c360c6a31f3a5b4687d Oct 30 05:54:37.626772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.625395 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5"] Oct 30 05:54:37.815969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.815940 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:54:37.816639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.816238 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:54:37.816639 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.816610 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 05:54:37.821957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.821929 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:37.822601 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.822580 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:37.823400 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:37.823379 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:54:38.307610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.306861 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:54:38.307610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.307581 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:54:38.307874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.307631 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:54:38.509629 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.509581 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" event={"ID":"6a5fb421-9f59-458f-8516-32ce09653507","Type":"ContainerStarted","Data":"e4dc20453fd5a0252c8d1f8cd53c268bb82f459afa4f45182f4e6384aecab40e"} Oct 30 05:54:38.509816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.509655 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" event={"ID":"6a5fb421-9f59-458f-8516-32ce09653507","Type":"ContainerStarted","Data":"c162abf356a356e23c659137dd488d5c632488a160569c360c6a31f3a5b4687d"} Oct 30 05:54:38.572603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.553073 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" podStartSLOduration=2.553053101 podStartE2EDuration="2.553053101s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:54:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:54:38.551557567 +0000 UTC m=+1385.127244618" watchObservedRunningTime="2026-10-30 05:54:38.553053101 +0000 UTC m=+1385.128740157" Oct 30 05:54:38.800085 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.800047 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:54:38.800409 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:38.800365 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 05:54:39.044474 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.044409 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7"] Oct 30 05:54:39.049146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.049121 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-eab7e800-2qpn7"] Oct 30 05:54:39.318922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.316782 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:54:39.318922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.316844 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:54:39.521307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.521274 5565 generic.go:356] "Generic (PLEG): container finished" podID="6a5fb421-9f59-458f-8516-32ce09653507" containerID="e4dc20453fd5a0252c8d1f8cd53c268bb82f459afa4f45182f4e6384aecab40e" exitCode=0 Oct 30 05:54:39.521498 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.521434 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" event={"ID":"6a5fb421-9f59-458f-8516-32ce09653507","Type":"ContainerDied","Data":"e4dc20453fd5a0252c8d1f8cd53c268bb82f459afa4f45182f4e6384aecab40e"} Oct 30 05:54:39.611745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.611663 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d49aa9be-9cab-427e-89bb-c347cd359102" path="/var/lib/kubelet/pods/d49aa9be-9cab-427e-89bb-c347cd359102/volumes" Oct 30 05:54:39.810398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.810355 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:54:39.810620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:39.810428 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 05:54:40.306569 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:40.305452 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:54:40.306569 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:40.305517 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:54:40.813607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:40.799784 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:54:40.813607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:40.799845 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 05:54:40.920334 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:40.920267 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:41.016269 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.004093 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca\") pod \"6a5fb421-9f59-458f-8516-32ce09653507\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " Oct 30 05:54:41.016269 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.004391 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca\") pod \"6a5fb421-9f59-458f-8516-32ce09653507\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " Oct 30 05:54:41.016269 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.004642 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8\" (UniqueName: \"kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8\") pod \"6a5fb421-9f59-458f-8516-32ce09653507\" (UID: \"6a5fb421-9f59-458f-8516-32ce09653507\") " Oct 30 05:54:41.016269 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.015498 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca" pod "6a5fb421-9f59-458f-8516-32ce09653507" (UID: "6a5fb421-9f59-458f-8516-32ce09653507"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:41.017818 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.017193 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:41.022409 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.022381 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8" pod "6a5fb421-9f59-458f-8516-32ce09653507" (UID: "6a5fb421-9f59-458f-8516-32ce09653507"). InnerVolumeSpecName "kube-api-access-nlgx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:41.022738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.022709 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca" pod "6a5fb421-9f59-458f-8516-32ce09653507" (UID: "6a5fb421-9f59-458f-8516-32ce09653507"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:54:41.118489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.118454 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlgx8\" (UniqueName: \"kubernetes.io/projected/6a5fb421-9f59-458f-8516-32ce09653507-kube-api-access-nlgx8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:41.118489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.118482 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a5fb421-9f59-458f-8516-32ce09653507-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:41.309404 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.309365 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:54:41.309967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.309928 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:54:41.615951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.615879 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" event={"ID":"6a5fb421-9f59-458f-8516-32ce09653507","Type":"ContainerDied","Data":"c162abf356a356e23c659137dd488d5c632488a160569c360c6a31f3a5b4687d"} Oct 30 05:54:41.616154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.616141 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c162abf356a356e23c659137dd488d5c632488a160569c360c6a31f3a5b4687d" Oct 30 05:54:41.616229 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.615909 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5" Oct 30 05:54:41.800550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.799939 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:54:41.800550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:41.799999 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 05:54:42.305809 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:42.305764 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" start-of-body= Oct 30 05:54:42.306035 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:42.305842 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.145:15021/healthz/ready\": dial tcp 10.134.0.145:15021: connect: connection refused" Oct 30 05:54:42.327605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:42.327569 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:54:42.799148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:42.799102 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" start-of-body= Oct 30 05:54:42.799360 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:42.799167 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.138:15021/healthz/ready\": dial tcp 10.134.0.138:15021: connect: connection refused" Oct 30 05:54:43.310278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.310251 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:54:43.310585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.310558 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:54:43.311170 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.311151 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 05:54:43.335350 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.335328 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:54:43.805253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.805221 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:54:43.806170 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.806144 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:54:43.809005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:43.808979 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 05:54:46.035743 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:46.035711 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2"] Oct 30 05:54:46.047725 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:46.047699 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-97f3dd70-2xvn2"] Oct 30 05:54:47.230690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:47.230649 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw"] Oct 30 05:54:47.287376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:47.287346 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54"] Oct 30 05:54:47.574913 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:54:47.574874 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:54:47.577776 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:47.577749 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc8d83c9-c5ad-40a9-8953-189df1ec4027" path="/var/lib/kubelet/pods/dc8d83c9-c5ad-40a9-8953-189df1ec4027/volumes" Oct 30 05:54:50.037425 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:50.037391 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk"] Oct 30 05:54:50.049339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:50.049311 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30c12092-tpmmk"] Oct 30 05:54:51.037731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:51.037699 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz"] Oct 30 05:54:51.050076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:51.050046 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f72470f6-7g2cz"] Oct 30 05:54:51.575039 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:51.575003 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fa5a0ec-e74f-4085-af8f-37fb28865655" path="/var/lib/kubelet/pods/7fa5a0ec-e74f-4085-af8f-37fb28865655/volumes" Oct 30 05:54:51.575626 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:51.575612 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb1675da-0614-4a54-bf5b-cd387988f3f8" path="/var/lib/kubelet/pods/fb1675da-0614-4a54-bf5b-cd387988f3f8/volumes" Oct 30 05:54:52.231233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.231174 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" podUID="faae66a6-23c5-49f8-96a0-ccb809394308" containerName="payload-processing" containerID="cri-o://9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa" gracePeriod=25 Oct 30 05:54:52.288605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.288562 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" podUID="676d796e-7f82-43f3-8fc6-9207c20f4d36" containerName="payload-pre-processing" containerID="cri-o://34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac" gracePeriod=25 Oct 30 05:54:52.628184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.628152 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:54:52.696623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.696593 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:54:52.721523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.721491 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj\" (UniqueName: \"kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj\") pod \"676d796e-7f82-43f3-8fc6-9207c20f4d36\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " Oct 30 05:54:52.721722 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.721709 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume\") pod \"676d796e-7f82-43f3-8fc6-9207c20f4d36\" (UID: \"676d796e-7f82-43f3-8fc6-9207c20f4d36\") " Oct 30 05:54:52.721781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.721762 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume\") pod \"faae66a6-23c5-49f8-96a0-ccb809394308\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " Oct 30 05:54:52.721837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.721804 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8\" (UniqueName: \"kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8\") pod \"faae66a6-23c5-49f8-96a0-ccb809394308\" (UID: \"faae66a6-23c5-49f8-96a0-ccb809394308\") " Oct 30 05:54:52.722112 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.722077 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume" pod "676d796e-7f82-43f3-8fc6-9207c20f4d36" (UID: "676d796e-7f82-43f3-8fc6-9207c20f4d36"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:52.722247 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.722205 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume" pod "faae66a6-23c5-49f8-96a0-ccb809394308" (UID: "faae66a6-23c5-49f8-96a0-ccb809394308"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:54:52.722621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.722601 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/676d796e-7f82-43f3-8fc6-9207c20f4d36-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:52.722621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.722623 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faae66a6-23c5-49f8-96a0-ccb809394308-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:52.724195 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.724173 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8" pod "faae66a6-23c5-49f8-96a0-ccb809394308" (UID: "faae66a6-23c5-49f8-96a0-ccb809394308"). InnerVolumeSpecName "kube-api-access-bjhg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:52.724293 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.724279 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj" pod "676d796e-7f82-43f3-8fc6-9207c20f4d36" (UID: "676d796e-7f82-43f3-8fc6-9207c20f4d36"). InnerVolumeSpecName "kube-api-access-ff5vj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:54:52.750150 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.750118 5565 generic.go:356] "Generic (PLEG): container finished" podID="faae66a6-23c5-49f8-96a0-ccb809394308" containerID="9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa" exitCode=0 Oct 30 05:54:52.750328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.750177 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" Oct 30 05:54:52.750328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.750231 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" event={"ID":"faae66a6-23c5-49f8-96a0-ccb809394308","Type":"ContainerDied","Data":"9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa"} Oct 30 05:54:52.750472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.750385 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw" event={"ID":"faae66a6-23c5-49f8-96a0-ccb809394308","Type":"ContainerDied","Data":"f51911a2df2e9143db5a716cb84e7ba6d9523cec4093ec2226728ac204343431"} Oct 30 05:54:52.750472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.750457 5565 scope.go:122] "RemoveContainer" containerID="9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa" Oct 30 05:54:52.752614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.752593 5565 generic.go:356] "Generic (PLEG): container finished" podID="676d796e-7f82-43f3-8fc6-9207c20f4d36" containerID="34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac" exitCode=0 Oct 30 05:54:52.752815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.752655 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" Oct 30 05:54:52.753008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.752668 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" event={"ID":"676d796e-7f82-43f3-8fc6-9207c20f4d36","Type":"ContainerDied","Data":"34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac"} Oct 30 05:54:52.753089 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.753042 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54" event={"ID":"676d796e-7f82-43f3-8fc6-9207c20f4d36","Type":"ContainerDied","Data":"a3f0ce1399286596f57372610f6a2e6dab18ed720509d2afc821d3702df4dcb2"} Oct 30 05:54:52.772037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.772014 5565 scope.go:122] "RemoveContainer" containerID="9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa" Oct 30 05:54:52.774965 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:54:52.774933 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa\": container with ID starting with 9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa not found: ID does not exist" containerID="9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa" Oct 30 05:54:52.775083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.774968 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa"} err="failed to get container status \"9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa\": rpc error: code = NotFound desc = could not find container \"9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa\": container with ID starting with 9905cf3b98d12ab78aa0950b859266e051084b70d8b5d3c2d3c6c577b66bc1fa not found: ID does not exist" Oct 30 05:54:52.775083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.774992 5565 scope.go:122] "RemoveContainer" containerID="34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac" Oct 30 05:54:52.788276 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.788252 5565 scope.go:122] "RemoveContainer" containerID="34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac" Oct 30 05:54:52.788588 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:54:52.788568 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac\": container with ID starting with 34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac not found: ID does not exist" containerID="34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac" Oct 30 05:54:52.788659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.788594 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac"} err="failed to get container status \"34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac\": rpc error: code = NotFound desc = could not find container \"34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac\": container with ID starting with 34f9a6a2dc3bac52fb1e77060eacc0f139c4f8fef4e8f6b6a6454ea62bea5aac not found: ID does not exist" Oct 30 05:54:52.824179 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.824113 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff5vj\" (UniqueName: \"kubernetes.io/projected/676d796e-7f82-43f3-8fc6-9207c20f4d36-kube-api-access-ff5vj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:52.824179 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.824138 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjhg8\" (UniqueName: \"kubernetes.io/projected/faae66a6-23c5-49f8-96a0-ccb809394308-kube-api-access-bjhg8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:54:52.831299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.831275 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54"] Oct 30 05:54:52.836316 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.836295 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-8bab43d7-86df8c47c5-x2m54"] Oct 30 05:54:52.871570 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.871544 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw"] Oct 30 05:54:52.875711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:52.875692 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-8bab43d7-97b78f99d-fhnxw"] Oct 30 05:54:53.570852 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:53.570781 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="676d796e-7f82-43f3-8fc6-9207c20f4d36" path="/var/lib/kubelet/pods/676d796e-7f82-43f3-8fc6-9207c20f4d36/volumes" Oct 30 05:54:53.571242 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:53.571227 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faae66a6-23c5-49f8-96a0-ccb809394308" path="/var/lib/kubelet/pods/faae66a6-23c5-49f8-96a0-ccb809394308/volumes" Oct 30 05:54:54.683572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:54.683540 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5"] Oct 30 05:54:54.695708 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:54.695679 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-8bab43d7-trlv5"] Oct 30 05:54:55.569244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:55.569211 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a5fb421-9f59-458f-8516-32ce09653507" path="/var/lib/kubelet/pods/6a5fb421-9f59-458f-8516-32ce09653507/volumes" Oct 30 05:54:58.033068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:58.033033 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc"] Oct 30 05:54:58.045365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:58.045335 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ca4b71f7-v49pc"] Oct 30 05:54:59.580104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:54:59.580071 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1dd564c-e34f-4595-93e7-7d58ce9d7102" path="/var/lib/kubelet/pods/a1dd564c-e34f-4595-93e7-7d58ce9d7102/volumes" Oct 30 05:55:00.536348 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.536318 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z"] Oct 30 05:55:00.537160 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537142 5565 cpu_manager.go:414] "containerMap: removing container" podUID="faae66a6-23c5-49f8-96a0-ccb809394308" containerName="payload-processing" Oct 30 05:55:00.537224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537163 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faae66a6-23c5-49f8-96a0-ccb809394308" containerName="payload-processing" Oct 30 05:55:00.537224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537182 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6a5fb421-9f59-458f-8516-32ce09653507" containerName="revoke-keys" Oct 30 05:55:00.537224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537188 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a5fb421-9f59-458f-8516-32ce09653507" containerName="revoke-keys" Oct 30 05:55:00.537224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537220 5565 cpu_manager.go:414] "containerMap: removing container" podUID="676d796e-7f82-43f3-8fc6-9207c20f4d36" containerName="payload-pre-processing" Oct 30 05:55:00.537349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537225 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="676d796e-7f82-43f3-8fc6-9207c20f4d36" containerName="payload-pre-processing" Oct 30 05:55:00.537349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537344 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a5fb421-9f59-458f-8516-32ce09653507" containerName="revoke-keys" Oct 30 05:55:00.537411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537358 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="676d796e-7f82-43f3-8fc6-9207c20f4d36" containerName="payload-pre-processing" Oct 30 05:55:00.537411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.537369 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="faae66a6-23c5-49f8-96a0-ccb809394308" containerName="payload-processing" Oct 30 05:55:00.541574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.541518 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.557111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.557084 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:55:00.584769 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.584743 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:55:00.590254 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.590222 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z"] Oct 30 05:55:00.689706 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.689673 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.689922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.689899 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4vnr\" (UniqueName: \"kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.690122 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.690100 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.791707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.791638 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.791838 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.791731 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.791936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.791920 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4vnr\" (UniqueName: \"kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.792386 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.792364 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.793952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.793935 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.816044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.816020 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4vnr\" (UniqueName: \"kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr\") pod \"maas-api-revoke-sub-386c4e11-qmq9z\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:00.856111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:00.856087 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:01.031726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.031690 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778"] Oct 30 05:55:01.041643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.041603 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9187914a-56778"] Oct 30 05:55:01.224144 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:01.224113 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3605ca7_fb50_435a_9000_8f8fe0246273.slice/crio-83dac70050155069722bcb3ddadcf55aa399a70b12fb8cf4dde9b2d91f92dc87 WatchSource:0}: Error finding container 83dac70050155069722bcb3ddadcf55aa399a70b12fb8cf4dde9b2d91f92dc87: Status 404 returned error can't find the container with id 83dac70050155069722bcb3ddadcf55aa399a70b12fb8cf4dde9b2d91f92dc87 Oct 30 05:55:01.224900 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.224873 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z"] Oct 30 05:55:01.570397 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.570371 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53a67de-159d-4406-9138-1dd8c088fcab" path="/var/lib/kubelet/pods/a53a67de-159d-4406-9138-1dd8c088fcab/volumes" Oct 30 05:55:01.888842 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.888808 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" event={"ID":"f3605ca7-fb50-435a-9000-8f8fe0246273","Type":"ContainerStarted","Data":"681008156ac074b27b88dfda9f0bf82fdea639d56021f1a9d9274e6cd5273437"} Oct 30 05:55:01.889268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.888867 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" event={"ID":"f3605ca7-fb50-435a-9000-8f8fe0246273","Type":"ContainerStarted","Data":"83dac70050155069722bcb3ddadcf55aa399a70b12fb8cf4dde9b2d91f92dc87"} Oct 30 05:55:01.936104 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:01.936052 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" podStartSLOduration=1.9360399689999999 podStartE2EDuration="1.936039969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:01.930401281 +0000 UTC m=+1408.506088336" watchObservedRunningTime="2026-10-30 05:55:01.936039969 +0000 UTC m=+1408.511727050" Oct 30 05:55:02.903508 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:02.903469 5565 generic.go:356] "Generic (PLEG): container finished" podID="f3605ca7-fb50-435a-9000-8f8fe0246273" containerID="681008156ac074b27b88dfda9f0bf82fdea639d56021f1a9d9274e6cd5273437" exitCode=0 Oct 30 05:55:02.903919 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:02.903564 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" event={"ID":"f3605ca7-fb50-435a-9000-8f8fe0246273","Type":"ContainerDied","Data":"681008156ac074b27b88dfda9f0bf82fdea639d56021f1a9d9274e6cd5273437"} Oct 30 05:55:04.119640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.119614 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:04.252174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.252145 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca\") pod \"f3605ca7-fb50-435a-9000-8f8fe0246273\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " Oct 30 05:55:04.252376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.252233 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr\" (UniqueName: \"kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr\") pod \"f3605ca7-fb50-435a-9000-8f8fe0246273\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " Oct 30 05:55:04.252376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.252289 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca\") pod \"f3605ca7-fb50-435a-9000-8f8fe0246273\" (UID: \"f3605ca7-fb50-435a-9000-8f8fe0246273\") " Oct 30 05:55:04.252769 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.252731 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca" pod "f3605ca7-fb50-435a-9000-8f8fe0246273" (UID: "f3605ca7-fb50-435a-9000-8f8fe0246273"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:04.253195 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.253170 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:04.254117 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.254091 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca" pod "f3605ca7-fb50-435a-9000-8f8fe0246273" (UID: "f3605ca7-fb50-435a-9000-8f8fe0246273"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:55:04.254390 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.254365 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr" pod "f3605ca7-fb50-435a-9000-8f8fe0246273" (UID: "f3605ca7-fb50-435a-9000-8f8fe0246273"). InnerVolumeSpecName "kube-api-access-l4vnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:04.359550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.355836 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3605ca7-fb50-435a-9000-8f8fe0246273-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:04.363399 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.363356 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4vnr\" (UniqueName: \"kubernetes.io/projected/f3605ca7-fb50-435a-9000-8f8fe0246273-kube-api-access-l4vnr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:04.560606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.558364 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz"] Oct 30 05:55:04.574548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.563498 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f3605ca7-fb50-435a-9000-8f8fe0246273" containerName="revoke-keys" Oct 30 05:55:04.574548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.563544 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3605ca7-fb50-435a-9000-8f8fe0246273" containerName="revoke-keys" Oct 30 05:55:04.574548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.563857 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3605ca7-fb50-435a-9000-8f8fe0246273" containerName="revoke-keys" Oct 30 05:55:04.597263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.589955 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz"] Oct 30 05:55:04.597263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.590987 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.634550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.625166 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-b3e625-openshift-default-dockercfg-xfmdc\"" Oct 30 05:55:04.700559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700301 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.700559 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700545 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.700815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700638 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.700815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700691 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.700815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700741 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.700815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700790 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.701045 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.700940 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dctkt\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.701045 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.701126 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.701315 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.701296 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.802931 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.802882 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.803129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803031 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.803129 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803062 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803375 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803476 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803544 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803596 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803638 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803690 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dctkt\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803793 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803874 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803922 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.803964 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.804709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.804121 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.806665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.806060 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.806665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.806308 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.829746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.829661 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.829746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.829678 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dctkt\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt\") pod \"e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:04.927427 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.927393 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" event={"ID":"f3605ca7-fb50-435a-9000-8f8fe0246273","Type":"ContainerDied","Data":"83dac70050155069722bcb3ddadcf55aa399a70b12fb8cf4dde9b2d91f92dc87"} Oct 30 05:55:04.927594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.927449 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83dac70050155069722bcb3ddadcf55aa399a70b12fb8cf4dde9b2d91f92dc87" Oct 30 05:55:04.927594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:04.927466 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z" Oct 30 05:55:05.033362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.033249 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:05.045551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.042226 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp"] Oct 30 05:55:05.052510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.052474 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4db310dc-2kgbp"] Oct 30 05:55:05.402192 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:05.402158 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de9a4c3_0b40_40f1_8f06_518a5382c642.slice/crio-e0b2fae70e1917a07ee7b5900a19becb7b41bf54ef0ce971e48956f613732989 WatchSource:0}: Error finding container e0b2fae70e1917a07ee7b5900a19becb7b41bf54ef0ce971e48956f613732989: Status 404 returned error can't find the container with id e0b2fae70e1917a07ee7b5900a19becb7b41bf54ef0ce971e48956f613732989 Oct 30 05:55:05.402552 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.402459 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz"] Oct 30 05:55:05.404977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.404941 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:05.405078 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.405015 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:05.405078 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.405060 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:05.581555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.577203 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d53c8aad-c181-402c-9dcd-0d4302d35bd2" path="/var/lib/kubelet/pods/d53c8aad-c181-402c-9dcd-0d4302d35bd2/volumes" Oct 30 05:55:05.939616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.939564 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" event={"ID":"1de9a4c3-0b40-40f1-8f06-518a5382c642","Type":"ContainerStarted","Data":"212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba"} Oct 30 05:55:05.939794 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.939673 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" event={"ID":"1de9a4c3-0b40-40f1-8f06-518a5382c642","Type":"ContainerStarted","Data":"e0b2fae70e1917a07ee7b5900a19becb7b41bf54ef0ce971e48956f613732989"} Oct 30 05:55:05.991208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:05.991160 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podStartSLOduration=1.991148411 podStartE2EDuration="1.991148411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:05.987129417 +0000 UTC m=+1412.562816482" watchObservedRunningTime="2026-10-30 05:55:05.991148411 +0000 UTC m=+1412.566835533" Oct 30 05:55:06.034207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:06.034182 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:06.040707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:06.040677 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:06.948686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:06.948652 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:06.949432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:06.949409 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:55:09.077133 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:09.077104 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m"] Oct 30 05:55:09.111185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:09.111151 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-426f7b1e-8pd8m"] Oct 30 05:55:09.570664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:09.570631 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf73b12d-1ea0-4542-ab67-69e662133fee" path="/var/lib/kubelet/pods/cf73b12d-1ea0-4542-ab67-69e662133fee/volumes" Oct 30 05:55:10.415505 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.415471 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk"] Oct 30 05:55:10.421473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.421453 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.443678 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.443650 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk"] Oct 30 05:55:10.467756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.450479 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-b3e625\"" Oct 30 05:55:10.467756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.450669 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-b3e625-dockercfg-qbtgp\"" Oct 30 05:55:10.467756 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.465667 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825"] Oct 30 05:55:10.533551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.523123 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.533551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.523385 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr\") pod \"payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.592649 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.579894 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825"] Oct 30 05:55:10.617647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.584662 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:10.635655 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.626705 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.656286 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.656249 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr\") pod \"payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.657007 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.631083 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.727913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.727470 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr\") pod \"payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.736283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.736194 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:10.758930 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.758897 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4mw6\" (UniqueName: \"kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6\") pod \"payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:10.759198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.759180 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:10.865063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.864345 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:10.865063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.864583 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4mw6\" (UniqueName: \"kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6\") pod \"payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:10.866575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.865637 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:10.889740 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:10.889712 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4mw6\" (UniqueName: \"kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6\") pod \"payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:11.056742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:11.056709 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:11.087906 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:11.087874 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543fd875_5b9d_481d_8746_20aa656d38f5.slice/crio-cd486ad40db46bd540f60f36403cb53b3526e3b82b4a404c3d83012bb75d9323 WatchSource:0}: Error finding container cd486ad40db46bd540f60f36403cb53b3526e3b82b4a404c3d83012bb75d9323: Status 404 returned error can't find the container with id cd486ad40db46bd540f60f36403cb53b3526e3b82b4a404c3d83012bb75d9323 Oct 30 05:55:11.089402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:11.089375 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk"] Oct 30 05:55:11.382117 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:11.382089 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7fabd8_1055_4ab4_8ab5_c37a54eb2b6f.slice/crio-daacfe036c7432bb36720dd795ab6bab861ff89c632e8679ada58b4c4ed75548 WatchSource:0}: Error finding container daacfe036c7432bb36720dd795ab6bab861ff89c632e8679ada58b4c4ed75548: Status 404 returned error can't find the container with id daacfe036c7432bb36720dd795ab6bab861ff89c632e8679ada58b4c4ed75548 Oct 30 05:55:11.383067 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:11.383039 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825"] Oct 30 05:55:11.998656 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:11.998604 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" event={"ID":"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f","Type":"ContainerStarted","Data":"747d28c8de081f6658fb96b77181711b54eb5760e7cc32bbac2532a539abb732"} Oct 30 05:55:11.999086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:11.998700 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" event={"ID":"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f","Type":"ContainerStarted","Data":"daacfe036c7432bb36720dd795ab6bab861ff89c632e8679ada58b4c4ed75548"} Oct 30 05:55:11.999086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:11.998845 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:12.000319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:12.000294 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" event={"ID":"543fd875-5b9d-481d-8746-20aa656d38f5","Type":"ContainerStarted","Data":"130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681"} Oct 30 05:55:12.000445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:12.000343 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:12.000445 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:12.000368 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" event={"ID":"543fd875-5b9d-481d-8746-20aa656d38f5","Type":"ContainerStarted","Data":"cd486ad40db46bd540f60f36403cb53b3526e3b82b4a404c3d83012bb75d9323"} Oct 30 05:55:12.044609 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:12.044564 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" podStartSLOduration=2.044549123 podStartE2EDuration="2.044549123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:12.043014951 +0000 UTC m=+1418.618702010" watchObservedRunningTime="2026-10-30 05:55:12.044549123 +0000 UTC m=+1418.620236174" Oct 30 05:55:12.087673 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:12.087628 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" podStartSLOduration=2.087615276 podStartE2EDuration="2.087615276s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:12.08540635 +0000 UTC m=+1418.661093418" watchObservedRunningTime="2026-10-30 05:55:12.087615276 +0000 UTC m=+1418.663302330" Oct 30 05:55:14.327551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:14.320033 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:55:17.546400 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:55:17.546353 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:55:18.736696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:18.736661 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml"] Oct 30 05:55:18.737239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:18.736842 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" containerID="cri-o://2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10" gracePeriod=30 Oct 30 05:55:20.035962 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:20.035926 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl"] Oct 30 05:55:20.048302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:20.048264 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-23daeeca-jbbwl"] Oct 30 05:55:21.569896 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:21.569864 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8222e6f0-536f-43b1-aecf-238ca0a1e03b" path="/var/lib/kubelet/pods/8222e6f0-536f-43b1-aecf-238ca0a1e03b/volumes" Oct 30 05:55:23.013098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:23.013064 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 05:55:23.013496 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:23.013148 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 05:55:24.026909 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.026884 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:55:24.140064 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140036 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140220 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140119 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140220 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140176 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140219 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140239 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140425 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140365 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140425 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140396 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140511 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140491 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:24.140746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140602 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:24.140746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140615 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:24.140746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140665 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy\") pod \"271c4540-a7c9-4694-aec5-3f3326a35d92\" (UID: \"271c4540-a7c9-4694-aec5-3f3326a35d92\") " Oct 30 05:55:24.140912 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140882 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:24.140970 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.140947 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:24.141605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.141580 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.141721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.141608 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.141721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.141623 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.141721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.141638 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/271c4540-a7c9-4694-aec5-3f3326a35d92-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.141721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.141652 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.142422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.142397 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:24.142868 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.142846 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:55:24.143085 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.143056 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:55:24.143208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.143153 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h" pod "271c4540-a7c9-4694-aec5-3f3326a35d92" (UID: "271c4540-a7c9-4694-aec5-3f3326a35d92"). InnerVolumeSpecName "kube-api-access-f8c5h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:24.194068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.194040 5565 generic.go:356] "Generic (PLEG): container finished" podID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerID="2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10" exitCode=0 Oct 30 05:55:24.194178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.194092 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" Oct 30 05:55:24.194178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.194116 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" event={"ID":"271c4540-a7c9-4694-aec5-3f3326a35d92","Type":"ContainerDied","Data":"2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10"} Oct 30 05:55:24.194178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.194167 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml" event={"ID":"271c4540-a7c9-4694-aec5-3f3326a35d92","Type":"ContainerDied","Data":"e4e513d01f8ced2177aba6e9cf0088b01e36c39094163466edbda5fcfdceae92"} Oct 30 05:55:24.194280 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.194190 5565 scope.go:122] "RemoveContainer" containerID="2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10" Oct 30 05:55:24.206140 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.206115 5565 scope.go:122] "RemoveContainer" containerID="2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10" Oct 30 05:55:24.206426 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:55:24.206409 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10\": container with ID starting with 2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10 not found: ID does not exist" containerID="2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10" Oct 30 05:55:24.206479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.206433 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10"} err="failed to get container status \"2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10\": rpc error: code = NotFound desc = could not find container \"2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10\": container with ID starting with 2cf53d31290a88afc6d9eb197be6626a7b428c9efcbeaccdd331ef6135887c10 not found: ID does not exist" Oct 30 05:55:24.243436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.243402 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f8c5h\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-kube-api-access-f8c5h\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.243436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.243434 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.243724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.243449 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.243724 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.243460 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/271c4540-a7c9-4694-aec5-3f3326a35d92-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:24.267142 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.267115 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml"] Oct 30 05:55:24.274000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:24.273975 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-7hcml"] Oct 30 05:55:25.068755 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.068716 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws"] Oct 30 05:55:25.073883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.073857 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6b0c89d9-lw8ws"] Oct 30 05:55:25.374207 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.374126 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc"] Oct 30 05:55:25.375006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.374987 5565 cpu_manager.go:414] "containerMap: removing container" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" Oct 30 05:55:25.375076 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.375007 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" Oct 30 05:55:25.375148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.375138 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" containerName="istio-proxy" Oct 30 05:55:25.380044 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.380026 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.405713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.405684 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-530a5e-dockercfg-t485t\"" Oct 30 05:55:25.405968 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.405938 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-530a5e\"" Oct 30 05:55:25.408423 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.408397 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc"] Oct 30 05:55:25.655643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.651364 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.655643 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.651587 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjkbn\" (UniqueName: \"kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn\") pod \"payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.684550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.683979 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="271c4540-a7c9-4694-aec5-3f3326a35d92" path="/var/lib/kubelet/pods/271c4540-a7c9-4694-aec5-3f3326a35d92/volumes" Oct 30 05:55:25.701590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.684951 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf" path="/var/lib/kubelet/pods/757ec86f-6b5b-479c-b7fb-c7ae9f44f2bf/volumes" Oct 30 05:55:25.701590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.685427 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8"] Oct 30 05:55:25.718894 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.718859 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8"] Oct 30 05:55:25.719083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.719004 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.755842 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.755802 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.756023 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.755919 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-530a5e-94d87598-krxs8\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.756143 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.756123 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-872md\" (UniqueName: \"kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md\") pod \"payload-processing-e2e-api-b-530a5e-94d87598-krxs8\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.756213 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.756183 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjkbn\" (UniqueName: \"kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn\") pod \"payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.756825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.756797 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.783618 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.783035 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjkbn\" (UniqueName: \"kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn\") pod \"payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:25.857722 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.857689 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-872md\" (UniqueName: \"kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md\") pod \"payload-processing-e2e-api-b-530a5e-94d87598-krxs8\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.858051 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.858034 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-530a5e-94d87598-krxs8\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.858750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.858731 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume\") pod \"payload-processing-e2e-api-b-530a5e-94d87598-krxs8\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.882123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.882096 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-872md\" (UniqueName: \"kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md\") pod \"payload-processing-e2e-api-b-530a5e-94d87598-krxs8\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:25.994472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:25.994389 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:26.057616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:26.057583 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:26.340760 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:26.340727 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc"] Oct 30 05:55:26.341194 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:26.340856 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff3ac10_bdb7_4868_9538_04113c5b677e.slice/crio-c9abbc8497ab358a82e3e243ffbca9739e2c6a8356541733a8ded5db641dd46c WatchSource:0}: Error finding container c9abbc8497ab358a82e3e243ffbca9739e2c6a8356541733a8ded5db641dd46c: Status 404 returned error can't find the container with id c9abbc8497ab358a82e3e243ffbca9739e2c6a8356541733a8ded5db641dd46c Oct 30 05:55:26.396872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:26.396845 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8"] Oct 30 05:55:26.397252 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:26.397208 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5695cff4_f0dd_4f1a_9a44_86e614677b48.slice/crio-4dc556fbab7ae97b11bf4a55e3e452bbb766e209360079a22de9902bcf7d048e WatchSource:0}: Error finding container 4dc556fbab7ae97b11bf4a55e3e452bbb766e209360079a22de9902bcf7d048e: Status 404 returned error can't find the container with id 4dc556fbab7ae97b11bf4a55e3e452bbb766e209360079a22de9902bcf7d048e Oct 30 05:55:27.231654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.231614 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" event={"ID":"6ff3ac10-bdb7-4868-9538-04113c5b677e","Type":"ContainerStarted","Data":"1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3"} Oct 30 05:55:27.231854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.231675 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" event={"ID":"6ff3ac10-bdb7-4868-9538-04113c5b677e","Type":"ContainerStarted","Data":"c9abbc8497ab358a82e3e243ffbca9739e2c6a8356541733a8ded5db641dd46c"} Oct 30 05:55:27.231854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.231746 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:27.233514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.233487 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" event={"ID":"5695cff4-f0dd-4f1a-9a44-86e614677b48","Type":"ContainerStarted","Data":"94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4"} Oct 30 05:55:27.233688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.233540 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" event={"ID":"5695cff4-f0dd-4f1a-9a44-86e614677b48","Type":"ContainerStarted","Data":"4dc556fbab7ae97b11bf4a55e3e452bbb766e209360079a22de9902bcf7d048e"} Oct 30 05:55:27.233688 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.233584 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:27.275594 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.275514 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" podStartSLOduration=2.275502239 podStartE2EDuration="2.275502239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:27.273077493 +0000 UTC m=+1433.848764547" watchObservedRunningTime="2026-10-30 05:55:27.275502239 +0000 UTC m=+1433.851189293" Oct 30 05:55:27.322139 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:27.322096 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" podStartSLOduration=2.322084895 podStartE2EDuration="2.322084895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:27.321094439 +0000 UTC m=+1433.896781490" watchObservedRunningTime="2026-10-30 05:55:27.322084895 +0000 UTC m=+1433.897771949" Oct 30 05:55:30.134093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.134058 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h"] Oct 30 05:55:30.139302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.139279 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.174584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.163413 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h"] Oct 30 05:55:30.289549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.281687 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:55:30.289549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.281843 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:55:30.329554 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.291214 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctlst\" (UniqueName: \"kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.336574 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.336513 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.337194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.337158 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.439568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.439428 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctlst\" (UniqueName: \"kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.439746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.439589 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.439746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.439627 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.440317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.440297 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.442394 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.442367 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.464091 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.464065 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctlst\" (UniqueName: \"kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:30.756606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:30.756518 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:31.102692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.102595 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx"] Oct 30 05:55:31.110077 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.110049 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.124187 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:31.124159 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef689c7b_fda8_4589_968c_54af2bf6ddcd.slice/crio-73bb538792aa76dbd066ca66e73f92bd99e3aef335948c21f5a4e58d923d460e WatchSource:0}: Error finding container 73bb538792aa76dbd066ca66e73f92bd99e3aef335948c21f5a4e58d923d460e: Status 404 returned error can't find the container with id 73bb538792aa76dbd066ca66e73f92bd99e3aef335948c21f5a4e58d923d460e Oct 30 05:55:31.124633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.124609 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h"] Oct 30 05:55:31.147330 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.147312 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-3abe42-openshift-default-dockercfg-6sv77\"" Oct 30 05:55:31.165575 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.165545 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx"] Oct 30 05:55:31.256705 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.256668 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.256860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.256773 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.256860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.256810 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.256860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.256836 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.256992 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.256875 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.257086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.257010 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvbvz\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.257086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.257064 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.257167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.257118 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.257203 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.257170 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360083 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.359984 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360441 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360508 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360555 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360579 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360621 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360669 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvbvz\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360703 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.360825 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.360723 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.364037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.363728 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.364037 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.364008 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.364224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.364133 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.364488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.364450 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.365056 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.365033 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.366602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.366517 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.368917 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.367722 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.373843 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.373820 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" event={"ID":"ef689c7b-fda8-4589-968c-54af2bf6ddcd","Type":"ContainerStarted","Data":"73bb538792aa76dbd066ca66e73f92bd99e3aef335948c21f5a4e58d923d460e"} Oct 30 05:55:31.387359 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.387332 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.388927 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.388904 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvbvz\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz\") pod \"e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.427994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.427959 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:31.779769 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.779744 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx"] Oct 30 05:55:31.780998 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:31.780971 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bee2e5b_a4b9_462f_bf9d_8c4cc1ffcf9f.slice/crio-58c8b7e685aa39f52f35f2e240db88d0835992d8ee072a48f3ac6d00469a264f WatchSource:0}: Error finding container 58c8b7e685aa39f52f35f2e240db88d0835992d8ee072a48f3ac6d00469a264f: Status 404 returned error can't find the container with id 58c8b7e685aa39f52f35f2e240db88d0835992d8ee072a48f3ac6d00469a264f Oct 30 05:55:31.783728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.783696 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:31.783820 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.783760 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:31.783820 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:31.783790 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:32.091560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.091016 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp"] Oct 30 05:55:32.150553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.148734 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm"] Oct 30 05:55:32.150553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.149042 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.198665 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.198634 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-df544d-openshift-default-dockercfg-6dwdl\"" Oct 30 05:55:32.291376 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.291342 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp"] Oct 30 05:55:32.291523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.291394 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm"] Oct 30 05:55:32.291523 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.291477 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.306010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.305478 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.305587 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.305645 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.305726 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.305779 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6svmp\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.305941 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.306052 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.306119 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.306517 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.306410 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.392447 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.392392 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" event={"ID":"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f","Type":"ContainerStarted","Data":"9c2e05b232a3aed282498bae4c364ca15ab565a32e7d53f643cc3e175f93ea4c"} Oct 30 05:55:32.392685 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.392487 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" event={"ID":"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f","Type":"ContainerStarted","Data":"58c8b7e685aa39f52f35f2e240db88d0835992d8ee072a48f3ac6d00469a264f"} Oct 30 05:55:32.394726 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.394691 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" event={"ID":"ef689c7b-fda8-4589-968c-54af2bf6ddcd","Type":"ContainerStarted","Data":"88b535f5d61ab7770a7ebacc1f4fed906f3747aaa61a5e38756c77e436d1d9f2"} Oct 30 05:55:32.408506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408478 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.408654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408604 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.408654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408633 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.408786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408675 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.408786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408731 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.408786 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408758 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.408935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408804 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.408935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408836 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.408935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408888 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6svmp\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.408935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.408886 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.409134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.409019 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.409134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.409114 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.410035 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410000 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.410155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410067 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.410155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410096 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.410155 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410139 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.411328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410576 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.411328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410648 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.411328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410850 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.411592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.410192 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.411682 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.411625 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.411682 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.411661 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.411956 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.411928 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fb7c\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.412249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.412228 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.424262 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.417989 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.429617 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.428213 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:32.430918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.430884 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:55:32.431016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.430941 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:55:32.434198 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.434172 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6svmp\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.435781 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.435762 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-d7jqp\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.449154 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.449098 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podStartSLOduration=1.449079832 podStartE2EDuration="1.449079832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:32.441687285 +0000 UTC m=+1439.017374339" watchObservedRunningTime="2026-10-30 05:55:32.449079832 +0000 UTC m=+1439.024766887" Oct 30 05:55:32.492302 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.492248 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" podStartSLOduration=2.492233298 podStartE2EDuration="2.492233298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:32.487251925 +0000 UTC m=+1439.062938979" watchObservedRunningTime="2026-10-30 05:55:32.492233298 +0000 UTC m=+1439.067920351" Oct 30 05:55:32.514250 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514214 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.514733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514581 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.514733 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514636 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.514928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514752 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.514928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514788 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.514928 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514855 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fb7c\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.515092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.514959 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.515092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515051 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.515211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515108 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.515211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515152 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.515745 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515723 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.515914 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515893 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.516008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515776 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.516008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.515742 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.517235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.517210 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.518773 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.518752 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.535506 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.535480 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:32.539300 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.539277 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.539520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.539498 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fb7c\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c\") pod \"e2e-api-a-df544d-openshift-default-bdd854567-8wbgm\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.611091 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.611032 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:32.936821 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:32.936790 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod943f3183_3c07_464e_980d_69b02e7a1a8f.slice/crio-51c50b418129e771345506af94cae98ee7fee2b598bf4a9fd19d45c4177b600c WatchSource:0}: Error finding container 51c50b418129e771345506af94cae98ee7fee2b598bf4a9fd19d45c4177b600c: Status 404 returned error can't find the container with id 51c50b418129e771345506af94cae98ee7fee2b598bf4a9fd19d45c4177b600c Oct 30 05:55:32.937537 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.937500 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp"] Oct 30 05:55:32.940058 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.940026 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:32.940168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.940096 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:32.940168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.940147 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:32.980459 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:32.980428 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2748ccfa_1021_40ac_9d37_ba329b658eba.slice/crio-7c2f2b8fc2feae5b2d1f0a939a6c5e071d918f78c365c8ea65ffbd60beecfa90 WatchSource:0}: Error finding container 7c2f2b8fc2feae5b2d1f0a939a6c5e071d918f78c365c8ea65ffbd60beecfa90: Status 404 returned error can't find the container with id 7c2f2b8fc2feae5b2d1f0a939a6c5e071d918f78c365c8ea65ffbd60beecfa90 Oct 30 05:55:32.981237 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.981217 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm"] Oct 30 05:55:32.983879 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.983847 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:32.983990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.983911 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:32.983990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:32.983942 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:55:33.407450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.407402 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" event={"ID":"943f3183-3c07-464e-980d-69b02e7a1a8f","Type":"ContainerStarted","Data":"332faf3279356de198e308f13c75ee70eaec591ebd4374ee63882ab8b8492f98"} Oct 30 05:55:33.407944 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.407473 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" event={"ID":"943f3183-3c07-464e-980d-69b02e7a1a8f","Type":"ContainerStarted","Data":"51c50b418129e771345506af94cae98ee7fee2b598bf4a9fd19d45c4177b600c"} Oct 30 05:55:33.410089 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.410058 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" event={"ID":"2748ccfa-1021-40ac-9d37-ba329b658eba","Type":"ContainerStarted","Data":"1fffa6b1c0f8e0c69015fab7bd045db2187418c6c42ea5f5914053e98d9c9fea"} Oct 30 05:55:33.410308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.410283 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" event={"ID":"2748ccfa-1021-40ac-9d37-ba329b658eba","Type":"ContainerStarted","Data":"7c2f2b8fc2feae5b2d1f0a939a6c5e071d918f78c365c8ea65ffbd60beecfa90"} Oct 30 05:55:33.412283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.412255 5565 generic.go:356] "Generic (PLEG): container finished" podID="ef689c7b-fda8-4589-968c-54af2bf6ddcd" containerID="88b535f5d61ab7770a7ebacc1f4fed906f3747aaa61a5e38756c77e436d1d9f2" exitCode=0 Oct 30 05:55:33.412413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.412342 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" event={"ID":"ef689c7b-fda8-4589-968c-54af2bf6ddcd","Type":"ContainerDied","Data":"88b535f5d61ab7770a7ebacc1f4fed906f3747aaa61a5e38756c77e436d1d9f2"} Oct 30 05:55:33.433631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.433605 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:33.460548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.456926 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podStartSLOduration=1.456906869 podStartE2EDuration="1.456906869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:33.454326575 +0000 UTC m=+1440.030013655" watchObservedRunningTime="2026-10-30 05:55:33.456906869 +0000 UTC m=+1440.032593925" Oct 30 05:55:33.504343 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.504296 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podStartSLOduration=1.5042822139999998 podStartE2EDuration="1.504282214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:33.502136846 +0000 UTC m=+1440.077823899" watchObservedRunningTime="2026-10-30 05:55:33.504282214 +0000 UTC m=+1440.079969268" Oct 30 05:55:33.535717 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.535685 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:33.611512 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:33.611484 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:34.426054 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.426021 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:34.427126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.427098 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:55:34.537452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.537211 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:55:34.537452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.537266 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 05:55:34.635102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.624874 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 05:55:34.635102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.624944 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 05:55:34.799578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.796455 5565 scope.go:122] "RemoveContainer" containerID="77b657c107609ce9b639f7a80b1c28a76aa9455f723979934bf9cca01a161ced" Oct 30 05:55:34.822587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.822543 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:34.828911 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.828882 5565 scope.go:122] "RemoveContainer" containerID="6552b12d22d4cc056375db924815dec572b44dc35cd6e01e16103ea2813ac2b6" Oct 30 05:55:34.840362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.840334 5565 scope.go:122] "RemoveContainer" containerID="8f20b5a5d1ca34c53aca3a61d4441bf61b23cde004611fef58760ff5479d2e35" Oct 30 05:55:34.850365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.850258 5565 scope.go:122] "RemoveContainer" containerID="6e548dda7ede91d9b8f21afb51dff47b28511b0487d8b38f4672a5912727757b" Oct 30 05:55:34.934904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.925998 5565 scope.go:122] "RemoveContainer" containerID="204ff350ed5eff73363aa5b017801f0212e86e81328f6869e509d68c8438c723" Oct 30 05:55:34.934904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.927271 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca\") pod \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " Oct 30 05:55:34.934904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.927399 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca\") pod \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " Oct 30 05:55:34.934904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.927465 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst\" (UniqueName: \"kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst\") pod \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\" (UID: \"ef689c7b-fda8-4589-968c-54af2bf6ddcd\") " Oct 30 05:55:34.934904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:34.929950 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca" pod "ef689c7b-fda8-4589-968c-54af2bf6ddcd" (UID: "ef689c7b-fda8-4589-968c-54af2bf6ddcd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:35.046553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.033674 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:35.046553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.034715 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst" pod "ef689c7b-fda8-4589-968c-54af2bf6ddcd" (UID: "ef689c7b-fda8-4589-968c-54af2bf6ddcd"). InnerVolumeSpecName "kube-api-access-ctlst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:35.158655 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.156713 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctlst\" (UniqueName: \"kubernetes.io/projected/ef689c7b-fda8-4589-968c-54af2bf6ddcd-kube-api-access-ctlst\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:35.242344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.238715 5565 scope.go:122] "RemoveContainer" containerID="ae87a1d2fa5473f980bbe9e610f61e7387eabaddb9222f95cb09aff74fdad68c" Oct 30 05:55:35.242344 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.239498 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca" pod "ef689c7b-fda8-4589-968c-54af2bf6ddcd" (UID: "ef689c7b-fda8-4589-968c-54af2bf6ddcd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:55:35.296641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.269620 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef689c7b-fda8-4589-968c-54af2bf6ddcd-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:35.296952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.296878 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf"] Oct 30 05:55:35.450597 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.441519 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aa7ce029-t8zjf"] Oct 30 05:55:35.570637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.568036 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:55:35.570637 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.568129 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 05:55:35.626871 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.626833 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 05:55:35.627069 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.626921 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 05:55:35.633791 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.632284 5565 scope.go:122] "RemoveContainer" containerID="cb8dee854f0998eafeae7b8e6c3881a15dfed63e3924dd3f22496f051394b370" Oct 30 05:55:35.737603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.737570 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4c094c9d-05cd-4697-bfed-b3b6c2a15ade" path="/var/lib/kubelet/pods/4c094c9d-05cd-4697-bfed-b3b6c2a15ade/volumes" Oct 30 05:55:35.743747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.743726 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" Oct 30 05:55:35.744132 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.743750 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h" event={"ID":"ef689c7b-fda8-4589-968c-54af2bf6ddcd","Type":"ContainerDied","Data":"73bb538792aa76dbd066ca66e73f92bd99e3aef335948c21f5a4e58d923d460e"} Oct 30 05:55:35.744235 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.744177 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="73bb538792aa76dbd066ca66e73f92bd99e3aef335948c21f5a4e58d923d460e" Oct 30 05:55:35.750664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:35.750649 5565 scope.go:122] "RemoveContainer" containerID="4fabdab30c71712b0dc9713728380e2edb3630865fd11b87bb187da5cd974bbd" Oct 30 05:55:36.095584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:36.075963 5565 scope.go:122] "RemoveContainer" containerID="d72e278259d8c2c34bb186146085122a66e79ed90ea1e9a95f912ce3c1a41e07" Oct 30 05:55:36.537194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:36.536798 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:55:36.537194 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:36.536866 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 05:55:36.614921 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:36.614871 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 05:55:36.615131 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:36.614939 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 05:55:37.443598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.443561 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj"] Oct 30 05:55:37.444434 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.444417 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ef689c7b-fda8-4589-968c-54af2bf6ddcd" containerName="revoke-keys" Oct 30 05:55:37.444494 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.444435 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef689c7b-fda8-4589-968c-54af2bf6ddcd" containerName="revoke-keys" Oct 30 05:55:37.444633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.444622 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef689c7b-fda8-4589-968c-54af2bf6ddcd" containerName="revoke-keys" Oct 30 05:55:37.452046 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.452021 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:37.479551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.468766 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-3abe42-dockercfg-qpv4x\"" Oct 30 05:55:37.479551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.471694 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-3abe42\"" Oct 30 05:55:37.512547 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.511606 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj"] Oct 30 05:55:37.538836 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.538800 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:55:37.539263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.538853 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 05:55:37.616853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.615312 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 05:55:37.616853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.615393 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 05:55:37.616853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.616330 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xwc\" (UniqueName: \"kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc\") pod \"payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:37.616853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.616460 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:37.736631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.729094 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xwc\" (UniqueName: \"kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc\") pod \"payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:37.736631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.729176 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:37.736631 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.730397 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:38.003604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:37.999631 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xwc\" (UniqueName: \"kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc\") pod \"payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:38.139514 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.139483 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:38.249998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.248871 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:55:38.249998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.249027 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:55:38.540485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.539841 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:55:38.540485 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.539898 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 05:55:38.611981 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.611894 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 05:55:38.611981 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.611952 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 05:55:38.881558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:38.876238 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj"] Oct 30 05:55:39.014551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:39.010891 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" event={"ID":"5d17330c-7e57-4aa1-b3b2-59301813307f","Type":"ContainerStarted","Data":"a716b7b0b6fd13f560daf478ac07d032c114237a79976933a76f05e1a77c9c8b"} Oct 30 05:55:39.537861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:39.536848 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 05:55:39.537861 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:39.536926 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 05:55:39.612314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:39.612016 5565 patch_prober.go:29] interesting pod/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 30 05:55:39.612314 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:39.612065 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 30 05:55:40.223557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.213087 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" event={"ID":"5d17330c-7e57-4aa1-b3b2-59301813307f","Type":"ContainerStarted","Data":"092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4"} Oct 30 05:55:40.223557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.214507 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:40.307561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.303343 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" podStartSLOduration=3.303322964 podStartE2EDuration="3.303322964s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:40.298611033 +0000 UTC m=+1446.874298091" watchObservedRunningTime="2026-10-30 05:55:40.303322964 +0000 UTC m=+1446.879010019" Oct 30 05:55:40.555188 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.555153 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:40.557275 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.557251 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:40.558711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.558664 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:55:40.703482 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.703429 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:40.704839 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.704808 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:40.705768 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:40.705747 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:55:41.140417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.140375 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw"] Oct 30 05:55:41.177063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.177027 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc"] Oct 30 05:55:41.601904 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.601864 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5"] Oct 30 05:55:41.608646 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.608619 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.611078 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.611049 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5"] Oct 30 05:55:41.625107 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.625080 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:55:41.625263 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.625248 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:55:41.652992 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.652959 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.653157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.653115 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d2vk\" (UniqueName: \"kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.653339 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.653278 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.755086 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.755053 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.755497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.755158 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d2vk\" (UniqueName: \"kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.755497 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.755293 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.755885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.755860 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.757905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.757876 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.781872 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.781845 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d2vk\" (UniqueName: \"kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk\") pod \"maas-api-revoke-sub-a49fcf92-m5fx5\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:41.928653 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:41.928563 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:42.268328 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:55:42.268296 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11f60d19_e41b_45d0_b527_7d87ad79f416.slice/crio-9d06d85fcb665f20e377aab15dd17020e4b63c02f2e1984b2088668e858900cd WatchSource:0}: Error finding container 9d06d85fcb665f20e377aab15dd17020e4b63c02f2e1984b2088668e858900cd: Status 404 returned error can't find the container with id 9d06d85fcb665f20e377aab15dd17020e4b63c02f2e1984b2088668e858900cd Oct 30 05:55:42.268883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:42.268852 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5"] Oct 30 05:55:43.274926 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:43.274879 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" event={"ID":"11f60d19-e41b-45d0-b527-7d87ad79f416","Type":"ContainerStarted","Data":"15513c429954fc47b16974d9db4c4043ac29657aa22c9ad2bbfe3f4651d8433b"} Oct 30 05:55:43.275436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:43.275009 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" event={"ID":"11f60d19-e41b-45d0-b527-7d87ad79f416","Type":"ContainerStarted","Data":"9d06d85fcb665f20e377aab15dd17020e4b63c02f2e1984b2088668e858900cd"} Oct 30 05:55:43.319063 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:43.319010 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" podStartSLOduration=2.318996531 podStartE2EDuration="2.318996531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:55:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:55:43.317141548 +0000 UTC m=+1449.892828600" watchObservedRunningTime="2026-10-30 05:55:43.318996531 +0000 UTC m=+1449.894683585" Oct 30 05:55:44.297761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:44.297729 5565 generic.go:356] "Generic (PLEG): container finished" podID="11f60d19-e41b-45d0-b527-7d87ad79f416" containerID="15513c429954fc47b16974d9db4c4043ac29657aa22c9ad2bbfe3f4651d8433b" exitCode=0 Oct 30 05:55:44.298224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:44.297827 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" event={"ID":"11f60d19-e41b-45d0-b527-7d87ad79f416","Type":"ContainerDied","Data":"15513c429954fc47b16974d9db4c4043ac29657aa22c9ad2bbfe3f4651d8433b"} Oct 30 05:55:45.486400 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.486365 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:45.528539 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.528493 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca\") pod \"11f60d19-e41b-45d0-b527-7d87ad79f416\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " Oct 30 05:55:45.528702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.528680 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca\") pod \"11f60d19-e41b-45d0-b527-7d87ad79f416\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " Oct 30 05:55:45.528984 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.528960 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk\" (UniqueName: \"kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk\") pod \"11f60d19-e41b-45d0-b527-7d87ad79f416\" (UID: \"11f60d19-e41b-45d0-b527-7d87ad79f416\") " Oct 30 05:55:45.529097 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.529065 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca" pod "11f60d19-e41b-45d0-b527-7d87ad79f416" (UID: "11f60d19-e41b-45d0-b527-7d87ad79f416"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:55:45.530004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.529978 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:45.531250 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.531211 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca" pod "11f60d19-e41b-45d0-b527-7d87ad79f416" (UID: "11f60d19-e41b-45d0-b527-7d87ad79f416"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:55:45.532903 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.532877 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk" pod "11f60d19-e41b-45d0-b527-7d87ad79f416" (UID: "11f60d19-e41b-45d0-b527-7d87ad79f416"). InnerVolumeSpecName "kube-api-access-8d2vk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:55:45.631744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.631712 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11f60d19-e41b-45d0-b527-7d87ad79f416-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:45.631744 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:45.631738 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d2vk\" (UniqueName: \"kubernetes.io/projected/11f60d19-e41b-45d0-b527-7d87ad79f416-kube-api-access-8d2vk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:55:46.140968 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:46.140927 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerName="payload-processing" containerID="cri-o://284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4" gracePeriod=25 Oct 30 05:55:46.177299 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:46.177263 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerName="payload-pre-processing" containerID="cri-o://9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16" gracePeriod=25 Oct 30 05:55:46.319193 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:46.319162 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" Oct 30 05:55:46.319338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:46.319189 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5" event={"ID":"11f60d19-e41b-45d0-b527-7d87ad79f416","Type":"ContainerDied","Data":"9d06d85fcb665f20e377aab15dd17020e4b63c02f2e1984b2088668e858900cd"} Oct 30 05:55:46.319338 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:46.319258 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d06d85fcb665f20e377aab15dd17020e4b63c02f2e1984b2088668e858900cd" Oct 30 05:55:47.534076 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:55:47.534037 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:55:52.266480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:52.266451 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 05:55:54.137633 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:54.137592 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:55:54.138118 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:55:54.137688 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:04.138865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:04.138823 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:04.139318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:04.139058 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 05:56:06.506167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.506132 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:56:06.555278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.555241 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:56:06.595345 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.595304 5565 generic.go:356] "Generic (PLEG): container finished" podID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerID="9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16" exitCode=0 Oct 30 05:56:06.595661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.595372 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" Oct 30 05:56:06.595661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.595404 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" event={"ID":"99c45e58-fe46-4c9f-9dfa-53be9a553aff","Type":"ContainerDied","Data":"9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16"} Oct 30 05:56:06.595661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.595480 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc" event={"ID":"99c45e58-fe46-4c9f-9dfa-53be9a553aff","Type":"ContainerDied","Data":"b042a159f68cdbaba79dba607063db88120bc45edd4406e52e22593b0bd326f2"} Oct 30 05:56:06.595661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.595514 5565 scope.go:122] "RemoveContainer" containerID="9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16" Oct 30 05:56:06.597892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.597858 5565 generic.go:356] "Generic (PLEG): container finished" podID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerID="284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4" exitCode=0 Oct 30 05:56:06.598019 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.597921 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" Oct 30 05:56:06.598019 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.597921 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" event={"ID":"6b832e8d-23d0-47e3-8f47-bae82bbc7e18","Type":"ContainerDied","Data":"284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4"} Oct 30 05:56:06.598111 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.598053 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw" event={"ID":"6b832e8d-23d0-47e3-8f47-bae82bbc7e18","Type":"ContainerDied","Data":"37ca1639cd426808c30484b059b1f4cc952955fe67e450c79e8df92e3b892146"} Oct 30 05:56:06.610994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.610924 5565 scope.go:122] "RemoveContainer" containerID="9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16" Oct 30 05:56:06.611445 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:56:06.611408 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16\": container with ID starting with 9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16 not found: ID does not exist" containerID="9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16" Oct 30 05:56:06.611667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.611460 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16"} err="failed to get container status \"9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16\": rpc error: code = NotFound desc = could not find container \"9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16\": container with ID starting with 9236c75d631c49d0c72d5ad333401bf189ff4eeba2d878aaf442909962c18f16 not found: ID does not exist" Oct 30 05:56:06.611667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.611490 5565 scope.go:122] "RemoveContainer" containerID="284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4" Oct 30 05:56:06.612182 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.612157 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume\") pod \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " Oct 30 05:56:06.612432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.612417 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782\" (UniqueName: \"kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782\") pod \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\" (UID: \"6b832e8d-23d0-47e3-8f47-bae82bbc7e18\") " Oct 30 05:56:06.612556 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.612519 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm\" (UniqueName: \"kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm\") pod \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " Oct 30 05:56:06.612709 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.612668 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume\") pod \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\" (UID: \"99c45e58-fe46-4c9f-9dfa-53be9a553aff\") " Oct 30 05:56:06.612788 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.612766 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume" pod "6b832e8d-23d0-47e3-8f47-bae82bbc7e18" (UID: "6b832e8d-23d0-47e3-8f47-bae82bbc7e18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:06.613831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.613804 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:06.615145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.615115 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume" pod "99c45e58-fe46-4c9f-9dfa-53be9a553aff" (UID: "99c45e58-fe46-4c9f-9dfa-53be9a553aff"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:06.615774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.615744 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm" pod "99c45e58-fe46-4c9f-9dfa-53be9a553aff" (UID: "99c45e58-fe46-4c9f-9dfa-53be9a553aff"). InnerVolumeSpecName "kube-api-access-8nmlm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:06.616318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.616290 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782" pod "6b832e8d-23d0-47e3-8f47-bae82bbc7e18" (UID: "6b832e8d-23d0-47e3-8f47-bae82bbc7e18"). InnerVolumeSpecName "kube-api-access-2c782". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:06.629738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.629705 5565 scope.go:122] "RemoveContainer" containerID="284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4" Oct 30 05:56:06.630216 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:56:06.630185 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4\": container with ID starting with 284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4 not found: ID does not exist" containerID="284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4" Oct 30 05:56:06.630317 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.630223 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4"} err="failed to get container status \"284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4\": rpc error: code = NotFound desc = could not find container \"284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4\": container with ID starting with 284a631f24b41ba89022d6a8211a9f754660fd61a7a0f6829d23213c3795b7d4 not found: ID does not exist" Oct 30 05:56:06.715616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.715495 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2c782\" (UniqueName: \"kubernetes.io/projected/6b832e8d-23d0-47e3-8f47-bae82bbc7e18-kube-api-access-2c782\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:06.715616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.715563 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nmlm\" (UniqueName: \"kubernetes.io/projected/99c45e58-fe46-4c9f-9dfa-53be9a553aff-kube-api-access-8nmlm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:06.715616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.715574 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99c45e58-fe46-4c9f-9dfa-53be9a553aff-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:06.876223 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:06.876179 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:56:07.034558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:07.033618 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc"] Oct 30 05:56:07.062551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:07.054855 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-d7lkc"] Oct 30 05:56:07.112587 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:07.110630 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw"] Oct 30 05:56:07.119553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:07.113401 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-snbmw"] Oct 30 05:56:07.571640 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:07.571597 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" path="/var/lib/kubelet/pods/6b832e8d-23d0-47e3-8f47-bae82bbc7e18/volumes" Oct 30 05:56:07.572239 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:07.572220 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" path="/var/lib/kubelet/pods/99c45e58-fe46-4c9f-9dfa-53be9a553aff/volumes" Oct 30 05:56:11.236550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.235691 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 05:56:11.276553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.237724 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 05:56:11.277886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.277840 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" probeResult="failure" output="Get \"https://10.134.0.98:8000/ready\": context deadline exceeded" Oct 30 05:56:11.298550 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.279197 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" probeResult="failure" output="Get \"https://10.134.0.45:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:11.330882 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.330845 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 05:56:11.367221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.336604 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:11.368202 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.354593 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 05:56:11.368659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.355322 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 05:56:11.380588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.355349 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.75:8000/ready\": context deadline exceeded" Oct 30 05:56:11.498827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.498725 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2" podUID="8025876f-d78e-4475-afd3-4ba65437847f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.27:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 05:56:11.628553 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:11.625225 5565 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 30 05:56:17.546253 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:56:17.546209 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:56:20.220479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:20.220398 5565 generic.go:356] "Generic (PLEG): container finished" podID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerID="212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba" exitCode=0 Oct 30 05:56:20.220897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:20.220469 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" event={"ID":"1de9a4c3-0b40-40f1-8f06-518a5382c642","Type":"ContainerDied","Data":"212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba"} Oct 30 05:56:20.220994 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:20.220979 5565 scope.go:122] "RemoveContainer" containerID="212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba" Oct 30 05:56:20.222949 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:20.222925 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:56:20.223001 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:20.222993 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:56:20.223050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:20.223024 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:56:21.263095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:21.259186 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" event={"ID":"1de9a4c3-0b40-40f1-8f06-518a5382c642","Type":"ContainerStarted","Data":"05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3"} Oct 30 05:56:21.948970 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:21.948934 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:56:22.034183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:22.034146 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:56:22.034380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:22.034205 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q"] Oct 30 05:56:22.035268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:22.035245 5565 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:56:22.035362 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:22.035302 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 05:56:22.048164 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:22.048133 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ed2e49-hcw4q"] Oct 30 05:56:23.040472 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:23.040429 5565 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:56:23.041216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:23.041179 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 05:56:23.637438 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:23.637404 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26c0ceba-2a05-4678-9d17-82dee819b5bb" path="/var/lib/kubelet/pods/26c0ceba-2a05-4678-9d17-82dee819b5bb/volumes" Oct 30 05:56:23.690450 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:23.690417 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h"] Oct 30 05:56:23.697318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:23.697289 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-z9j9h"] Oct 30 05:56:24.039490 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:24.039448 5565 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:56:24.039753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:24.039577 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 05:56:25.034465 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:25.034431 5565 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:56:25.034876 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:25.034478 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 05:56:25.589095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:25.589061 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef689c7b-fda8-4589-968c-54af2bf6ddcd" path="/var/lib/kubelet/pods/ef689c7b-fda8-4589-968c-54af2bf6ddcd/volumes" Oct 30 05:56:26.034515 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:26.034485 5565 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 05:56:26.034899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:26.034563 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 05:56:27.041295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:27.041249 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:56:27.042776 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:27.042751 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 05:56:30.036583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:30.036547 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx"] Oct 30 05:56:30.049814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:30.049783 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc38413d-ws9bx"] Oct 30 05:56:31.570606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:31.570571 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7366be08-ab4b-4d73-831a-7a0f081d582e" path="/var/lib/kubelet/pods/7366be08-ab4b-4d73-831a-7a0f081d582e/volumes" Oct 30 05:56:32.064857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.064819 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f"] Oct 30 05:56:32.078441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.078406 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dc8f378f-f9d7f"] Oct 30 05:56:32.593146 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.593111 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4"] Oct 30 05:56:32.594021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594000 5565 cpu_manager.go:414] "containerMap: removing container" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerName="payload-pre-processing" Oct 30 05:56:32.594080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594022 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerName="payload-pre-processing" Oct 30 05:56:32.594080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594046 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerName="payload-processing" Oct 30 05:56:32.594080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594052 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerName="payload-processing" Oct 30 05:56:32.594080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594062 5565 cpu_manager.go:414] "containerMap: removing container" podUID="11f60d19-e41b-45d0-b527-7d87ad79f416" containerName="revoke-keys" Oct 30 05:56:32.594080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594077 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11f60d19-e41b-45d0-b527-7d87ad79f416" containerName="revoke-keys" Oct 30 05:56:32.594244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594215 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="99c45e58-fe46-4c9f-9dfa-53be9a553aff" containerName="payload-pre-processing" Oct 30 05:56:32.594244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594226 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b832e8d-23d0-47e3-8f47-bae82bbc7e18" containerName="payload-processing" Oct 30 05:56:32.594244 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.594235 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="11f60d19-e41b-45d0-b527-7d87ad79f416" containerName="revoke-keys" Oct 30 05:56:32.599508 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.599479 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:32.619648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.619570 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-a1c17ebc\"" Oct 30 05:56:32.638817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.633682 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-a1c17ebc-dockercfg-qhh2h\"" Oct 30 05:56:32.638817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.635282 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:32.638817 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.635906 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh5bk\" (UniqueName: \"kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:32.642732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.642704 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4"] Oct 30 05:56:32.790972 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.774927 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh5bk\" (UniqueName: \"kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:32.825551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.805430 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:32.867602 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:32.806731 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:33.047125 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.046987 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr"] Oct 30 05:56:33.051098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.051067 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh5bk\" (UniqueName: \"kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk\") pod \"payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:33.053480 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.053457 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr"] Oct 30 05:56:33.053611 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.053574 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.208118 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.208034 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgq9\" (UniqueName: \"kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.208296 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.208222 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.215010 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.214988 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:33.312396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.311710 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.312396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.311945 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgq9\" (UniqueName: \"kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.312796 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.312771 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.352273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.352238 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgq9\" (UniqueName: \"kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9\") pod \"payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.378163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.378132 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:33.644785 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:56:33.644751 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf41c9de_39ef_4393_8bb6_b57ea97ecfb5.slice/crio-b3960389eecf1bdc70e34eb5b235a59bd927bb5e3b90ec0414e29e73cfa59ce1 WatchSource:0}: Error finding container b3960389eecf1bdc70e34eb5b235a59bd927bb5e3b90ec0414e29e73cfa59ce1: Status 404 returned error can't find the container with id b3960389eecf1bdc70e34eb5b235a59bd927bb5e3b90ec0414e29e73cfa59ce1 Oct 30 05:56:33.649881 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.649852 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc03814b-3458-4fb9-9b32-112310d3649e" path="/var/lib/kubelet/pods/fc03814b-3458-4fb9-9b32-112310d3649e/volumes" Oct 30 05:56:33.653103 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.652215 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4"] Oct 30 05:56:33.965827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:33.965798 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr"] Oct 30 05:56:33.966704 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:56:33.966670 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda88a121e_ec20_4ec3_9a50_9101abe41578.slice/crio-b023c095b83d75a867613cc48f045a4cfdb4c32105b95bb84c98dac2f4d0ac28 WatchSource:0}: Error finding container b023c095b83d75a867613cc48f045a4cfdb4c32105b95bb84c98dac2f4d0ac28: Status 404 returned error can't find the container with id b023c095b83d75a867613cc48f045a4cfdb4c32105b95bb84c98dac2f4d0ac28 Oct 30 05:56:34.591163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.591108 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" event={"ID":"a88a121e-ec20-4ec3-9a50-9101abe41578","Type":"ContainerStarted","Data":"a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6"} Oct 30 05:56:34.591380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.591186 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:34.591380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.591212 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" event={"ID":"a88a121e-ec20-4ec3-9a50-9101abe41578","Type":"ContainerStarted","Data":"b023c095b83d75a867613cc48f045a4cfdb4c32105b95bb84c98dac2f4d0ac28"} Oct 30 05:56:34.594815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.593469 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" event={"ID":"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5","Type":"ContainerStarted","Data":"e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e"} Oct 30 05:56:34.594815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.593550 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" event={"ID":"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5","Type":"ContainerStarted","Data":"b3960389eecf1bdc70e34eb5b235a59bd927bb5e3b90ec0414e29e73cfa59ce1"} Oct 30 05:56:34.594815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.593704 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:34.639844 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.639794 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" podStartSLOduration=2.639780539 podStartE2EDuration="2.639780539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:34.636247482 +0000 UTC m=+1501.211934536" watchObservedRunningTime="2026-10-30 05:56:34.639780539 +0000 UTC m=+1501.215467586" Oct 30 05:56:34.684614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:34.684550 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" podStartSLOduration=2.6845161490000002 podStartE2EDuration="2.684516149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:34.680943688 +0000 UTC m=+1501.256630960" watchObservedRunningTime="2026-10-30 05:56:34.684516149 +0000 UTC m=+1501.260203202" Oct 30 05:56:36.724748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.691710 5565 scope.go:122] "RemoveContainer" containerID="5286633b865b3079a8791c3355d6fadc6d30165159fd486faa6405468625d8ae" Oct 30 05:56:36.772793 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.771140 5565 scope.go:122] "RemoveContainer" containerID="d03a7adb43166500b02efe2675cca7ebf7576410076a3aaafd5f5fef7229cae0" Oct 30 05:56:36.795823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.791345 5565 scope.go:122] "RemoveContainer" containerID="3cf6ac0160396ae7b46fe107e4ad2c5fb06710336d34b171d2610a4944c71570" Oct 30 05:56:36.819792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.816295 5565 scope.go:122] "RemoveContainer" containerID="e7c0710edf9d827f1b7e2b09c0b8b1c01b2280e753395651e02d4b43582488e2" Oct 30 05:56:36.833068 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.833046 5565 scope.go:122] "RemoveContainer" containerID="7494312ccab1a22d11b97001cbffdfb88c25571fcacc550c3a5f36dd6b5fbbaa" Oct 30 05:56:36.845318 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.845294 5565 scope.go:122] "RemoveContainer" containerID="a880630fa4703a778ef623d88e91199be3783782bad2dcbc5bf41bab7c57f420" Oct 30 05:56:36.859220 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.859199 5565 scope.go:122] "RemoveContainer" containerID="92486031bcc481e7701f78cb3ad987caa2c433b69f063cd7a4147a7d33c16c62" Oct 30 05:56:36.871178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.871155 5565 scope.go:122] "RemoveContainer" containerID="f0d82eb5c5afa3a2eba5e5aae264a8786d9242f451d6b51ed6e17576fbefb5dc" Oct 30 05:56:36.881698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.881676 5565 scope.go:122] "RemoveContainer" containerID="d72dc5acfdd75ae9d20d4918ac884e072b0c191a2ef5f8aafce7e522d3da2360" Oct 30 05:56:36.891815 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.891795 5565 scope.go:122] "RemoveContainer" containerID="05acadc1cbcbd33fd3bfe2eed62b8d57030c45e3e2a117366e6fb859869a66fb" Oct 30 05:56:36.906794 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:36.903345 5565 scope.go:122] "RemoveContainer" containerID="0c5de13434f1d2581d2f8892a8d43abd4273d8eb512f56d7b73775fe3173bc02" Oct 30 05:56:38.033719 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:38.033680 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp"] Oct 30 05:56:38.046740 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:38.046715 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a95de3f5-l46fp"] Oct 30 05:56:39.576066 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:39.576028 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86712f48-16de-4189-88f8-59f6e2008bba" path="/var/lib/kubelet/pods/86712f48-16de-4189-88f8-59f6e2008bba/volumes" Oct 30 05:56:42.032041 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:42.031999 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s"] Oct 30 05:56:42.046336 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:42.046297 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-372e855c-f6r7s"] Oct 30 05:56:43.571563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:43.571511 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f69e09e9-44b7-4474-9f0e-eb93c259fe76" path="/var/lib/kubelet/pods/f69e09e9-44b7-4474-9f0e-eb93c259fe76/volumes" Oct 30 05:56:45.615342 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:45.615312 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:56:45.615865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:45.615385 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:56:47.537846 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:56:47.537808 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:56:48.675252 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:48.675216 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g"] Oct 30 05:56:48.675695 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:48.675402 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" containerID="cri-o://7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe" gracePeriod=30 Oct 30 05:56:54.002446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.002411 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:56:54.102676 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.102643 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103025 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103000 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103166 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103151 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103349 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103334 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103512 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103495 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103644 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103629 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103748 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103734 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.103813 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103770 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.105443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.103952 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs\") pod \"16768327-985e-4f85-9e4f-85d0eb1c191b\" (UID: \"16768327-985e-4f85-9e4f-85d0eb1c191b\") " Oct 30 05:56:54.105443 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.105016 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:54.107151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.106666 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:56:54.107151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.106745 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:54.107151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.106924 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:54.107151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.107116 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:54.116800 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.116767 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:56:54.116893 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.116834 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "kube-api-access-dfgs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:54.116940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.116910 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:56:54.118036 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.118009 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy" pod "16768327-985e-4f85-9e4f-85d0eb1c191b" (UID: "16768327-985e-4f85-9e4f-85d0eb1c191b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:56:54.142732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.142702 5565 generic.go:356] "Generic (PLEG): container finished" podID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerID="7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe" exitCode=0 Oct 30 05:56:54.142828 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.142809 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" Oct 30 05:56:54.142881 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.142834 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" event={"ID":"16768327-985e-4f85-9e4f-85d0eb1c191b","Type":"ContainerDied","Data":"7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe"} Oct 30 05:56:54.142925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.142892 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g" event={"ID":"16768327-985e-4f85-9e4f-85d0eb1c191b","Type":"ContainerDied","Data":"d5c810572dfdcf9dd156bbf0288361329d99a5a015f1f2e88885e6904a311ac5"} Oct 30 05:56:54.142966 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.142922 5565 scope.go:122] "RemoveContainer" containerID="7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe" Oct 30 05:56:54.155259 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.155235 5565 scope.go:122] "RemoveContainer" containerID="7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe" Oct 30 05:56:54.155582 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:56:54.155563 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe\": container with ID starting with 7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe not found: ID does not exist" containerID="7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe" Oct 30 05:56:54.155662 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.155590 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe"} err="failed to get container status \"7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe\": rpc error: code = NotFound desc = could not find container \"7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe\": container with ID starting with 7e326a987d53a34e9aee64130d056f5b83489d7c35d2128b45cb87fdf5a33ffe not found: ID does not exist" Oct 30 05:56:54.207479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207445 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207477 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207488 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfgs2\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-kube-api-access-dfgs2\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207500 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207513 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207613 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207633 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207647 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/16768327-985e-4f85-9e4f-85d0eb1c191b-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.207672 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/16768327-985e-4f85-9e4f-85d0eb1c191b-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:56:54.211750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.211681 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g"] Oct 30 05:56:54.218232 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:54.218195 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-d6x6g"] Oct 30 05:56:55.571725 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:55.571689 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" path="/var/lib/kubelet/pods/16768327-985e-4f85-9e4f-85d0eb1c191b/volumes" Oct 30 05:56:55.728080 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:55.728045 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:56:57.944579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.944516 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm"] Oct 30 05:56:57.945993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.945967 5565 cpu_manager.go:414] "containerMap: removing container" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" Oct 30 05:56:57.945993 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.945994 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" Oct 30 05:56:57.946205 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.946191 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="16768327-985e-4f85-9e4f-85d0eb1c191b" containerName="istio-proxy" Oct 30 05:56:57.950948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.950923 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:57.970711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.970681 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm"] Oct 30 05:56:57.977568 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.977495 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:56:57.977702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:57.977565 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:56:58.099449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.099406 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zs7tk\" (UniqueName: \"kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.099645 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.099620 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.099978 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.099950 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.202792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.202654 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.202997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.202816 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zs7tk\" (UniqueName: \"kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.202997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.202907 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.203687 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.203661 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.205572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.205513 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.229675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.229647 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zs7tk\" (UniqueName: \"kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk\") pod \"maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.268368 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.268046 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:56:58.598602 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:56:58.598567 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90c55ec4_ec7f_4600_82b7_fcb6b282a060.slice/crio-f6f7cba76d171042d087acbf938d20070c43dd85bbdcf581e62b696c890066ab WatchSource:0}: Error finding container f6f7cba76d171042d087acbf938d20070c43dd85bbdcf581e62b696c890066ab: Status 404 returned error can't find the container with id f6f7cba76d171042d087acbf938d20070c43dd85bbdcf581e62b696c890066ab Oct 30 05:56:58.599050 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:58.598972 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm"] Oct 30 05:56:59.205681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:59.205648 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" event={"ID":"90c55ec4-ec7f-4600-82b7-fcb6b282a060","Type":"ContainerStarted","Data":"5878c4813d410bf2c349aa75a6b507d8df26f0489c930282db8224972422701e"} Oct 30 05:56:59.206039 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:59.205700 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" event={"ID":"90c55ec4-ec7f-4600-82b7-fcb6b282a060","Type":"ContainerStarted","Data":"f6f7cba76d171042d087acbf938d20070c43dd85bbdcf581e62b696c890066ab"} Oct 30 05:56:59.250499 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:56:59.250454 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" podStartSLOduration=2.250440895 podStartE2EDuration="2.250440895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:56:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:56:59.249443587 +0000 UTC m=+1525.825130640" watchObservedRunningTime="2026-10-30 05:56:59.250440895 +0000 UTC m=+1525.826127949" Oct 30 05:57:00.217826 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:00.217730 5565 generic.go:356] "Generic (PLEG): container finished" podID="90c55ec4-ec7f-4600-82b7-fcb6b282a060" containerID="5878c4813d410bf2c349aa75a6b507d8df26f0489c930282db8224972422701e" exitCode=0 Oct 30 05:57:00.217826 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:00.217804 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" event={"ID":"90c55ec4-ec7f-4600-82b7-fcb6b282a060","Type":"ContainerDied","Data":"5878c4813d410bf2c349aa75a6b507d8df26f0489c930282db8224972422701e"} Oct 30 05:57:01.712899 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.712862 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:57:01.838664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.838566 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk\" (UniqueName: \"kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk\") pod \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " Oct 30 05:57:01.838847 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.838732 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca\") pod \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " Oct 30 05:57:01.839591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.838983 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca\") pod \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\" (UID: \"90c55ec4-ec7f-4600-82b7-fcb6b282a060\") " Oct 30 05:57:01.843607 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.840251 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca" pod "90c55ec4-ec7f-4600-82b7-fcb6b282a060" (UID: "90c55ec4-ec7f-4600-82b7-fcb6b282a060"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:01.850959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.850928 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca" pod "90c55ec4-ec7f-4600-82b7-fcb6b282a060" (UID: "90c55ec4-ec7f-4600-82b7-fcb6b282a060"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:57:01.852495 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.851814 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk" pod "90c55ec4-ec7f-4600-82b7-fcb6b282a060" (UID: "90c55ec4-ec7f-4600-82b7-fcb6b282a060"). InnerVolumeSpecName "kube-api-access-zs7tk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:01.941897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.941853 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:01.941897 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.941896 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zs7tk\" (UniqueName: \"kubernetes.io/projected/90c55ec4-ec7f-4600-82b7-fcb6b282a060-kube-api-access-zs7tk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:01.942162 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:01.941913 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/90c55ec4-ec7f-4600-82b7-fcb6b282a060-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:02.299442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:02.299408 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" Oct 30 05:57:02.299442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:02.299423 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm" event={"ID":"90c55ec4-ec7f-4600-82b7-fcb6b282a060","Type":"ContainerDied","Data":"f6f7cba76d171042d087acbf938d20070c43dd85bbdcf581e62b696c890066ab"} Oct 30 05:57:02.300013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:02.299492 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6f7cba76d171042d087acbf938d20070c43dd85bbdcf581e62b696c890066ab" Oct 30 05:57:05.466721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:05.466676 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 05:57:05.467157 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:05.466754 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 05:57:08.101007 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.100964 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv"] Oct 30 05:57:08.103479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.102426 5565 cpu_manager.go:414] "containerMap: removing container" podUID="90c55ec4-ec7f-4600-82b7-fcb6b282a060" containerName="revoke-keys" Oct 30 05:57:08.103479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.102449 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90c55ec4-ec7f-4600-82b7-fcb6b282a060" containerName="revoke-keys" Oct 30 05:57:08.103479 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.102702 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="90c55ec4-ec7f-4600-82b7-fcb6b282a060" containerName="revoke-keys" Oct 30 05:57:08.122591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.122551 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.128774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.128739 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv"] Oct 30 05:57:08.231990 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.141851 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-452a59-openshift-default-dockercfg-wgvvb\"" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.324321 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.324469 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.324726 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.324806 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.324884 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h68cg\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.324951 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.325004 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.325039 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.326416 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.325163 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.580671 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.569273 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.569465 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.602077 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.602384 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.602471 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.602569 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h68cg\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.602643 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.602948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.602869 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.606204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.603081 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.606204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.603858 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.615886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.615842 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.616225 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.615853 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.616361 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.615912 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.616476 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.616146 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.617714 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.617685 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.618948 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.618919 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.626350 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.626322 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.641119 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.641085 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h68cg\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:08.752333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:08.752296 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:09.166027 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.165985 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv"] Oct 30 05:57:09.188787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.185963 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:57:09.188787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.186079 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:57:09.188787 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.186126 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:57:09.306468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.303019 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2"] Oct 30 05:57:09.438243 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.438146 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2"] Oct 30 05:57:09.438652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.438628 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.499584 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.482763 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:57:09.539592 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.512406 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:57:09.642273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.642233 5565 generic.go:356] "Generic (PLEG): container finished" podID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerID="9c2e05b232a3aed282498bae4c364ca15ab565a32e7d53f643cc3e175f93ea4c" exitCode=0 Oct 30 05:57:09.642430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.642314 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" event={"ID":"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f","Type":"ContainerDied","Data":"9c2e05b232a3aed282498bae4c364ca15ab565a32e7d53f643cc3e175f93ea4c"} Oct 30 05:57:09.642857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.642831 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.643034 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.643007 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.643268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.643246 5565 scope.go:122] "RemoveContainer" containerID="9c2e05b232a3aed282498bae4c364ca15ab565a32e7d53f643cc3e175f93ea4c" Oct 30 05:57:09.643500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.643471 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p9jb\" (UniqueName: \"kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.645305 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.645281 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" event={"ID":"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0","Type":"ContainerStarted","Data":"6a077ac003f26bcbf1f98f25b15b2b243799b8981d3168b1121c7bd7fbd32ae8"} Oct 30 05:57:09.645483 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.645449 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:57:09.645596 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.645563 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:57:09.645670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.645619 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:57:09.759246 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.759204 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.759667 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.759641 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.761824 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.761795 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p9jb\" (UniqueName: \"kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.762572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.761234 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.803669 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.794055 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.866582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.842854 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p9jb\" (UniqueName: \"kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb\") pod \"maas-api-revoke-keys-e2e-api-a-df544d-2zhv2\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:09.891652 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:09.885665 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:10.299323 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.299288 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2"] Oct 30 05:57:10.299795 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:57:10.299292 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21571bc6_d4d6_40b7_a521_c91db6ec8cbd.slice/crio-158338f2811bbf14d6d6cda38f328d350392ffdf4b57be8cd18aa499982b14ea WatchSource:0}: Error finding container 158338f2811bbf14d6d6cda38f328d350392ffdf4b57be8cd18aa499982b14ea: Status 404 returned error can't find the container with id 158338f2811bbf14d6d6cda38f328d350392ffdf4b57be8cd18aa499982b14ea Oct 30 05:57:10.656940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.656902 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" event={"ID":"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0","Type":"ContainerStarted","Data":"aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4"} Oct 30 05:57:10.658591 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.658559 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" event={"ID":"21571bc6-d4d6-40b7-a521-c91db6ec8cbd","Type":"ContainerStarted","Data":"158338f2811bbf14d6d6cda38f328d350392ffdf4b57be8cd18aa499982b14ea"} Oct 30 05:57:10.660890 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.660861 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" event={"ID":"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f","Type":"ContainerStarted","Data":"7dc99a69d1a41e71c2c0959b848c7f06e5e9201e4ae2e735671aaadb3110d6d6"} Oct 30 05:57:10.709578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.709483 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podStartSLOduration=2.709464627 podStartE2EDuration="2.709464627s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:10.708623995 +0000 UTC m=+1537.284311049" watchObservedRunningTime="2026-10-30 05:57:10.709464627 +0000 UTC m=+1537.285151682" Oct 30 05:57:10.753398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.753363 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:10.760410 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:10.760380 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:11.428622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.428595 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:57:11.429092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.429069 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:57:11.429151 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.429132 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:57:11.686404 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.686318 5565 generic.go:356] "Generic (PLEG): container finished" podID="21571bc6-d4d6-40b7-a521-c91db6ec8cbd" containerID="9050be9726838af12724cd5f8e6a85dc46393b73005ba19d45a0f0f2cbe9b8bf" exitCode=0 Oct 30 05:57:11.686572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.686406 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" event={"ID":"21571bc6-d4d6-40b7-a521-c91db6ec8cbd","Type":"ContainerDied","Data":"9050be9726838af12724cd5f8e6a85dc46393b73005ba19d45a0f0f2cbe9b8bf"} Oct 30 05:57:11.686966 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.686948 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:11.687995 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:11.687978 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:57:12.082128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:12.082084 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c"] Oct 30 05:57:12.099335 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:12.099303 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-318cd5c3-wsh4c"] Oct 30 05:57:12.437274 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:12.437171 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:57:12.437274 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:12.437234 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:57:13.046072 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.046041 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:13.069555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.067254 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca\") pod \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " Oct 30 05:57:13.069555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.067460 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca\") pod \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " Oct 30 05:57:13.069555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.067645 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb\" (UniqueName: \"kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb\") pod \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\" (UID: \"21571bc6-d4d6-40b7-a521-c91db6ec8cbd\") " Oct 30 05:57:13.081980 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.079974 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca" pod "21571bc6-d4d6-40b7-a521-c91db6ec8cbd" (UID: "21571bc6-d4d6-40b7-a521-c91db6ec8cbd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:13.086114 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.086069 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb" pod "21571bc6-d4d6-40b7-a521-c91db6ec8cbd" (UID: "21571bc6-d4d6-40b7-a521-c91db6ec8cbd"). InnerVolumeSpecName "kube-api-access-7p9jb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:13.086500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.086474 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca" pod "21571bc6-d4d6-40b7-a521-c91db6ec8cbd" (UID: "21571bc6-d4d6-40b7-a521-c91db6ec8cbd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:57:13.185117 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.185062 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.185117 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.185109 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p9jb\" (UniqueName: \"kubernetes.io/projected/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-kube-api-access-7p9jb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.185396 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.185129 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/21571bc6-d4d6-40b7-a521-c91db6ec8cbd-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:13.428855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.428743 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:57:13.428855 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.428816 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:57:13.494822 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.494789 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l"] Oct 30 05:57:13.495681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.495663 5565 cpu_manager.go:414] "containerMap: removing container" podUID="21571bc6-d4d6-40b7-a521-c91db6ec8cbd" containerName="revoke-keys" Oct 30 05:57:13.495747 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.495682 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21571bc6-d4d6-40b7-a521-c91db6ec8cbd" containerName="revoke-keys" Oct 30 05:57:13.495853 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.495843 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="21571bc6-d4d6-40b7-a521-c91db6ec8cbd" containerName="revoke-keys" Oct 30 05:57:13.499906 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.499882 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.897801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.701632 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp4ff\" (UniqueName: \"kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.897801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.701780 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.897801 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.701899 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.949892 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.949850 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.950278 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.950252 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.950797 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.950769 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp4ff\" (UniqueName: \"kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.952442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.952412 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:13.988130 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.988091 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149" path="/var/lib/kubelet/pods/39dcf4f4-6dd1-4a7a-8f11-4ff3c26af149/volumes" Oct 30 05:57:13.989452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.989428 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l"] Oct 30 05:57:13.993341 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.993310 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" event={"ID":"21571bc6-d4d6-40b7-a521-c91db6ec8cbd","Type":"ContainerDied","Data":"158338f2811bbf14d6d6cda38f328d350392ffdf4b57be8cd18aa499982b14ea"} Oct 30 05:57:13.993427 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.993389 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="158338f2811bbf14d6d6cda38f328d350392ffdf4b57be8cd18aa499982b14ea" Oct 30 05:57:13.993772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.993748 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2" Oct 30 05:57:13.996183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:13.996159 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp4ff\" (UniqueName: \"kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:14.000166 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:14.000143 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:14.209387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:14.209296 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:14.429139 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:14.429111 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:57:14.429319 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:14.429161 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:57:14.581452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:14.581162 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l"] Oct 30 05:57:15.003144 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:15.003101 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" event={"ID":"b606f3c8-841d-4cf6-be9b-e323262fb52f","Type":"ContainerStarted","Data":"8f1932171c8315df932cf31c160e151006b3f814f4908f1d4f4e54d2d2e6528f"} Oct 30 05:57:15.428789 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:15.428755 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:57:15.428969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:15.428814 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:57:16.177670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:16.168690 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" event={"ID":"b606f3c8-841d-4cf6-be9b-e323262fb52f","Type":"ContainerStarted","Data":"5ba8428aec1e1da4c25e6fcbea489550bf90d361a938fbf006d280dfeda3bd56"} Oct 30 05:57:16.231862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:16.231794 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" podStartSLOduration=3.231776674 podStartE2EDuration="3.231776674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:57:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:57:16.224000247 +0000 UTC m=+1542.799687302" watchObservedRunningTime="2026-10-30 05:57:16.231776674 +0000 UTC m=+1542.807463737" Oct 30 05:57:16.428752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:16.428662 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 30 05:57:16.428752 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:16.428724 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 30 05:57:17.196949 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:17.196882 5565 generic.go:356] "Generic (PLEG): container finished" podID="b606f3c8-841d-4cf6-be9b-e323262fb52f" containerID="5ba8428aec1e1da4c25e6fcbea489550bf90d361a938fbf006d280dfeda3bd56" exitCode=0 Oct 30 05:57:17.197998 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:17.197022 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" event={"ID":"b606f3c8-841d-4cf6-be9b-e323262fb52f","Type":"ContainerDied","Data":"5ba8428aec1e1da4c25e6fcbea489550bf90d361a938fbf006d280dfeda3bd56"} Oct 30 05:57:17.434272 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:17.434242 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:57:17.434750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:17.434726 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:57:17.435564 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:17.435513 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 05:57:17.534799 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:57:17.534728 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:57:18.406863 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.406834 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:18.437018 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.436986 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff\" (UniqueName: \"kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff\") pod \"b606f3c8-841d-4cf6-be9b-e323262fb52f\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " Oct 30 05:57:18.437167 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.437031 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca\") pod \"b606f3c8-841d-4cf6-be9b-e323262fb52f\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " Oct 30 05:57:18.437273 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.437257 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca\") pod \"b606f3c8-841d-4cf6-be9b-e323262fb52f\" (UID: \"b606f3c8-841d-4cf6-be9b-e323262fb52f\") " Oct 30 05:57:18.437432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.437409 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca" pod "b606f3c8-841d-4cf6-be9b-e323262fb52f" (UID: "b606f3c8-841d-4cf6-be9b-e323262fb52f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:18.437828 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.437809 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:18.439101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.439078 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff" pod "b606f3c8-841d-4cf6-be9b-e323262fb52f" (UID: "b606f3c8-841d-4cf6-be9b-e323262fb52f"). InnerVolumeSpecName "kube-api-access-fp4ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:18.439169 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.439127 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca" pod "b606f3c8-841d-4cf6-be9b-e323262fb52f" (UID: "b606f3c8-841d-4cf6-be9b-e323262fb52f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:57:18.538999 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.538970 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b606f3c8-841d-4cf6-be9b-e323262fb52f-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:18.538999 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:18.538998 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fp4ff\" (UniqueName: \"kubernetes.io/projected/b606f3c8-841d-4cf6-be9b-e323262fb52f-kube-api-access-fp4ff\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:19.220379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:19.220336 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" Oct 30 05:57:19.220379 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:19.220365 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l" event={"ID":"b606f3c8-841d-4cf6-be9b-e323262fb52f","Type":"ContainerDied","Data":"8f1932171c8315df932cf31c160e151006b3f814f4908f1d4f4e54d2d2e6528f"} Oct 30 05:57:19.220648 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:19.220428 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8f1932171c8315df932cf31c160e151006b3f814f4908f1d4f4e54d2d2e6528f" Oct 30 05:57:23.883560 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:23.883500 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb"] Oct 30 05:57:23.919002 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:23.918971 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl"] Oct 30 05:57:28.884491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:28.884452 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" podUID="4bf047a4-573a-4f50-837f-2af39aeceb58" containerName="payload-processing" containerID="cri-o://7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7" gracePeriod=25 Oct 30 05:57:28.920164 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:28.920130 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" podUID="602f29b3-b904-4951-930a-b7b541ed6c9f" containerName="payload-pre-processing" containerID="cri-o://2f0155ea57daf7df990657d1ee17196f2d46465254f44da2889e4e1304a9ac5a" gracePeriod=25 Oct 30 05:57:29.288268 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.288232 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:57:29.330307 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.330275 5565 generic.go:356] "Generic (PLEG): container finished" podID="4bf047a4-573a-4f50-837f-2af39aeceb58" containerID="7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7" exitCode=0 Oct 30 05:57:29.330510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.330327 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" Oct 30 05:57:29.330510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.330355 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" event={"ID":"4bf047a4-573a-4f50-837f-2af39aeceb58","Type":"ContainerDied","Data":"7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7"} Oct 30 05:57:29.330510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.330415 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb" event={"ID":"4bf047a4-573a-4f50-837f-2af39aeceb58","Type":"ContainerDied","Data":"bded2aa16097d413cb943851fc1ed45de03d6dfb4c4ab30875448453ded6b141"} Oct 30 05:57:29.330510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.330443 5565 scope.go:122] "RemoveContainer" containerID="7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7" Oct 30 05:57:29.332296 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.332273 5565 generic.go:356] "Generic (PLEG): container finished" podID="602f29b3-b904-4951-930a-b7b541ed6c9f" containerID="2f0155ea57daf7df990657d1ee17196f2d46465254f44da2889e4e1304a9ac5a" exitCode=0 Oct 30 05:57:29.332619 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.332470 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" event={"ID":"602f29b3-b904-4951-930a-b7b541ed6c9f","Type":"ContainerDied","Data":"2f0155ea57daf7df990657d1ee17196f2d46465254f44da2889e4e1304a9ac5a"} Oct 30 05:57:29.352753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.350012 5565 scope.go:122] "RemoveContainer" containerID="7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7" Oct 30 05:57:29.352753 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:57:29.351698 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7\": container with ID starting with 7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7 not found: ID does not exist" containerID="7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7" Oct 30 05:57:29.352753 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.351730 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7"} err="failed to get container status \"7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7\": rpc error: code = NotFound desc = could not find container \"7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7\": container with ID starting with 7322900d91a3395204e5ca6919d91903ccd0d08ce34edcfb071499f5867805c7 not found: ID does not exist" Oct 30 05:57:29.380102 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.380072 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:57:29.476878 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.476798 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn\" (UniqueName: \"kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn\") pod \"4bf047a4-573a-4f50-837f-2af39aeceb58\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " Oct 30 05:57:29.477081 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.477068 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume\") pod \"602f29b3-b904-4951-930a-b7b541ed6c9f\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " Oct 30 05:57:29.477164 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.477149 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8\" (UniqueName: \"kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8\") pod \"602f29b3-b904-4951-930a-b7b541ed6c9f\" (UID: \"602f29b3-b904-4951-930a-b7b541ed6c9f\") " Oct 30 05:57:29.477233 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.477217 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume\") pod \"4bf047a4-573a-4f50-837f-2af39aeceb58\" (UID: \"4bf047a4-573a-4f50-837f-2af39aeceb58\") " Oct 30 05:57:29.477767 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.477501 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume" pod "602f29b3-b904-4951-930a-b7b541ed6c9f" (UID: "602f29b3-b904-4951-930a-b7b541ed6c9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:29.477935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.477905 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume" pod "4bf047a4-573a-4f50-837f-2af39aeceb58" (UID: "4bf047a4-573a-4f50-837f-2af39aeceb58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:57:29.478253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.478229 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/602f29b3-b904-4951-930a-b7b541ed6c9f-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:29.478253 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.478253 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bf047a4-573a-4f50-837f-2af39aeceb58-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:29.479430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.479403 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn" pod "4bf047a4-573a-4f50-837f-2af39aeceb58" (UID: "4bf047a4-573a-4f50-837f-2af39aeceb58"). InnerVolumeSpecName "kube-api-access-8hgxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:29.479759 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.479735 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8" pod "602f29b3-b904-4951-930a-b7b541ed6c9f" (UID: "602f29b3-b904-4951-930a-b7b541ed6c9f"). InnerVolumeSpecName "kube-api-access-xflh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:57:29.580432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.580407 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hgxn\" (UniqueName: \"kubernetes.io/projected/4bf047a4-573a-4f50-837f-2af39aeceb58-kube-api-access-8hgxn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:29.580557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.580434 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xflh8\" (UniqueName: \"kubernetes.io/projected/602f29b3-b904-4951-930a-b7b541ed6c9f-kube-api-access-xflh8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:57:29.676627 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.676598 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb"] Oct 30 05:57:29.680922 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:29.680900 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-e94bd8-7bc969f6db-tlmmb"] Oct 30 05:57:30.345214 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:30.345185 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" Oct 30 05:57:30.345711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:30.345185 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl" event={"ID":"602f29b3-b904-4951-930a-b7b541ed6c9f","Type":"ContainerDied","Data":"fd805a57d9efdd2f8adbab4085693ed427fcc6b241ab05386dc9363e3eb39b97"} Oct 30 05:57:30.345711 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:30.345318 5565 scope.go:122] "RemoveContainer" containerID="2f0155ea57daf7df990657d1ee17196f2d46465254f44da2889e4e1304a9ac5a" Oct 30 05:57:30.405168 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:30.405141 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl"] Oct 30 05:57:30.419021 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:30.418984 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-e94bd8-7b55cd8dcf-r6wsl"] Oct 30 05:57:31.569715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:31.569684 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bf047a4-573a-4f50-837f-2af39aeceb58" path="/var/lib/kubelet/pods/4bf047a4-573a-4f50-837f-2af39aeceb58/volumes" Oct 30 05:57:31.570183 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:31.570168 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602f29b3-b904-4951-930a-b7b541ed6c9f" path="/var/lib/kubelet/pods/602f29b3-b904-4951-930a-b7b541ed6c9f/volumes" Oct 30 05:57:37.033512 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:37.033472 5565 scope.go:122] "RemoveContainer" containerID="d9af6b6172e067b75464feb1980f54969566bcee09480cfe8b9c006c35c4de51" Oct 30 05:57:37.045216 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:37.045190 5565 scope.go:122] "RemoveContainer" containerID="6928a3a025e7d30787093222274bc60bb9f8b648dcf74689db6dd34a35e74a90" Oct 30 05:57:37.056725 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:37.056699 5565 scope.go:122] "RemoveContainer" containerID="c3f961618ff507cf68f33f336b7415c81439bd95efb866ae4b5b58f80292e6a1" Oct 30 05:57:37.068659 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:37.068634 5565 scope.go:122] "RemoveContainer" containerID="5974f16e31741aa11707cd5a2db15f460827d1acc61139ec353cba466032de3e" Oct 30 05:57:37.079446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:37.079421 5565 scope.go:122] "RemoveContainer" containerID="2bf5dcb02289c2306e4c6c182748adb0f96d5a0f7df76babf90ffa38d83adab6" Oct 30 05:57:37.090583 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:37.090514 5565 scope.go:122] "RemoveContainer" containerID="9b48576af5a5d2adfab624213c165c94e2d9034ee5eae5d355ae3dda5c11ff29" Oct 30 05:57:47.555947 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:57:47.555909 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:57:58.870283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.870247 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc"] Oct 30 05:57:58.871093 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871077 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b606f3c8-841d-4cf6-be9b-e323262fb52f" containerName="revoke-keys" Oct 30 05:57:58.871141 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871094 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b606f3c8-841d-4cf6-be9b-e323262fb52f" containerName="revoke-keys" Oct 30 05:57:58.871141 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871105 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4bf047a4-573a-4f50-837f-2af39aeceb58" containerName="payload-processing" Oct 30 05:57:58.871141 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871110 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bf047a4-573a-4f50-837f-2af39aeceb58" containerName="payload-processing" Oct 30 05:57:58.871141 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871117 5565 cpu_manager.go:414] "containerMap: removing container" podUID="602f29b3-b904-4951-930a-b7b541ed6c9f" containerName="payload-pre-processing" Oct 30 05:57:58.871141 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871122 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602f29b3-b904-4951-930a-b7b541ed6c9f" containerName="payload-pre-processing" Oct 30 05:57:58.871298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871246 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b606f3c8-841d-4cf6-be9b-e323262fb52f" containerName="revoke-keys" Oct 30 05:57:58.871298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871255 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bf047a4-573a-4f50-837f-2af39aeceb58" containerName="payload-processing" Oct 30 05:57:58.871298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.871263 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="602f29b3-b904-4951-930a-b7b541ed6c9f" containerName="payload-pre-processing" Oct 30 05:57:58.875226 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.875207 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:58.896808 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.896779 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc"] Oct 30 05:57:58.916582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.914806 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:57:58.916582 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.914932 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:57:58.994862 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.994828 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:58.995004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.994971 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmb46\" (UniqueName: \"kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:58.995217 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:58.995202 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.096219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.096191 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmb46\" (UniqueName: \"kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.096466 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.096447 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.096622 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.096584 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.097295 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.097270 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.098775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.098757 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.121569 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.121493 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmb46\" (UniqueName: \"kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46\") pod \"maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.190464 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.190431 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:57:59.520022 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.518100 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc"] Oct 30 05:57:59.526368 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:57:59.526328 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53750c8f_13b4_4bdc_9d00_db931538c3af.slice/crio-0591a3ace19326d8269fddedfe24a333542b0cd117c106679d45b4c449669bd1 WatchSource:0}: Error finding container 0591a3ace19326d8269fddedfe24a333542b0cd117c106679d45b4c449669bd1: Status 404 returned error can't find the container with id 0591a3ace19326d8269fddedfe24a333542b0cd117c106679d45b4c449669bd1 Oct 30 05:57:59.663085 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:57:59.663049 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" event={"ID":"53750c8f-13b4-4bdc-9d00-db931538c3af","Type":"ContainerStarted","Data":"0591a3ace19326d8269fddedfe24a333542b0cd117c106679d45b4c449669bd1"} Oct 30 05:58:00.673952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:00.673915 5565 generic.go:356] "Generic (PLEG): container finished" podID="53750c8f-13b4-4bdc-9d00-db931538c3af" containerID="4a01a3e3281f88861e4911a07c523421aafbe046b6efa4990401749741bcf10a" exitCode=0 Oct 30 05:58:00.674363 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:00.673997 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" event={"ID":"53750c8f-13b4-4bdc-9d00-db931538c3af","Type":"ContainerDied","Data":"4a01a3e3281f88861e4911a07c523421aafbe046b6efa4990401749741bcf10a"} Oct 30 05:58:01.863598 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.863571 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:58:01.950873 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.950782 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46\" (UniqueName: \"kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46\") pod \"53750c8f-13b4-4bdc-9d00-db931538c3af\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " Oct 30 05:58:01.951055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.950902 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca\") pod \"53750c8f-13b4-4bdc-9d00-db931538c3af\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " Oct 30 05:58:01.951124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.951072 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca\") pod \"53750c8f-13b4-4bdc-9d00-db931538c3af\" (UID: \"53750c8f-13b4-4bdc-9d00-db931538c3af\") " Oct 30 05:58:01.951389 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.951341 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca" pod "53750c8f-13b4-4bdc-9d00-db931538c3af" (UID: "53750c8f-13b4-4bdc-9d00-db931538c3af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:01.952100 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.952075 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:01.953677 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.953641 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46" pod "53750c8f-13b4-4bdc-9d00-db931538c3af" (UID: "53750c8f-13b4-4bdc-9d00-db931538c3af"). InnerVolumeSpecName "kube-api-access-mmb46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:01.954031 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:01.954002 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca" pod "53750c8f-13b4-4bdc-9d00-db931538c3af" (UID: "53750c8f-13b4-4bdc-9d00-db931538c3af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:02.053441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:02.053403 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53750c8f-13b4-4bdc-9d00-db931538c3af-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:02.053441 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:02.053438 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmb46\" (UniqueName: \"kubernetes.io/projected/53750c8f-13b4-4bdc-9d00-db931538c3af-kube-api-access-mmb46\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:02.691845 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:02.691816 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" Oct 30 05:58:02.692030 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:02.691814 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc" event={"ID":"53750c8f-13b4-4bdc-9d00-db931538c3af","Type":"ContainerDied","Data":"0591a3ace19326d8269fddedfe24a333542b0cd117c106679d45b4c449669bd1"} Oct 30 05:58:02.692030 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:02.691941 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0591a3ace19326d8269fddedfe24a333542b0cd117c106679d45b4c449669bd1" Oct 30 05:58:07.623002 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:07.622968 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:58:09.155784 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:09.155681 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr"] Oct 30 05:58:09.170917 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:09.167369 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4"] Oct 30 05:58:10.223176 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:10.223114 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm"] Oct 30 05:58:10.228694 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:10.228664 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-fd1c5d-bfxmm"] Oct 30 05:58:11.587291 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:11.587259 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90c55ec4-ec7f-4600-82b7-fcb6b282a060" path="/var/lib/kubelet/pods/90c55ec4-ec7f-4600-82b7-fcb6b282a060/volumes" Oct 30 05:58:14.171551 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.168263 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" podUID="cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" containerName="payload-pre-processing" containerID="cri-o://e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e" gracePeriod=25 Oct 30 05:58:14.172647 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.172584 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" podUID="a88a121e-ec20-4ec3-9a50-9101abe41578" containerName="payload-processing" containerID="cri-o://a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6" gracePeriod=25 Oct 30 05:58:14.543033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.542999 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:58:14.588215 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.588189 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:58:14.688029 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.687928 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume\") pod \"a88a121e-ec20-4ec3-9a50-9101abe41578\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " Oct 30 05:58:14.688555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.688425 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9\" (UniqueName: \"kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9\") pod \"a88a121e-ec20-4ec3-9a50-9101abe41578\" (UID: \"a88a121e-ec20-4ec3-9a50-9101abe41578\") " Oct 30 05:58:14.688555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.688494 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk\" (UniqueName: \"kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk\") pod \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " Oct 30 05:58:14.689773 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.688801 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume\") pod \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\" (UID: \"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5\") " Oct 30 05:58:14.689773 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.689002 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume" pod "a88a121e-ec20-4ec3-9a50-9101abe41578" (UID: "a88a121e-ec20-4ec3-9a50-9101abe41578"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:14.689970 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.689808 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume" pod "cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" (UID: "cf41c9de-39ef-4393-8bb6-b57ea97ecfb5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:14.690282 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.690255 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a88a121e-ec20-4ec3-9a50-9101abe41578-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:14.690365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.690290 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:14.691969 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.691943 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9" pod "a88a121e-ec20-4ec3-9a50-9101abe41578" (UID: "a88a121e-ec20-4ec3-9a50-9101abe41578"). InnerVolumeSpecName "kube-api-access-5jgq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:14.693723 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.693701 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk" pod "cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" (UID: "cf41c9de-39ef-4393-8bb6-b57ea97ecfb5"). InnerVolumeSpecName "kube-api-access-rh5bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:14.791977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.791942 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgq9\" (UniqueName: \"kubernetes.io/projected/a88a121e-ec20-4ec3-9a50-9101abe41578-kube-api-access-5jgq9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:14.791977 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.791973 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh5bk\" (UniqueName: \"kubernetes.io/projected/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5-kube-api-access-rh5bk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:14.925510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.925473 5565 generic.go:356] "Generic (PLEG): container finished" podID="cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" containerID="e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e" exitCode=0 Oct 30 05:58:14.925670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.925517 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" Oct 30 05:58:14.925670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.925558 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" event={"ID":"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5","Type":"ContainerDied","Data":"e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e"} Oct 30 05:58:14.925670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.925612 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4" event={"ID":"cf41c9de-39ef-4393-8bb6-b57ea97ecfb5","Type":"ContainerDied","Data":"b3960389eecf1bdc70e34eb5b235a59bd927bb5e3b90ec0414e29e73cfa59ce1"} Oct 30 05:58:14.925670 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.925634 5565 scope.go:122] "RemoveContainer" containerID="e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e" Oct 30 05:58:14.927411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.927387 5565 generic.go:356] "Generic (PLEG): container finished" podID="a88a121e-ec20-4ec3-9a50-9101abe41578" containerID="a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6" exitCode=0 Oct 30 05:58:14.927634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.927431 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" Oct 30 05:58:14.927634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.927449 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" event={"ID":"a88a121e-ec20-4ec3-9a50-9101abe41578","Type":"ContainerDied","Data":"a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6"} Oct 30 05:58:14.927634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.927480 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr" event={"ID":"a88a121e-ec20-4ec3-9a50-9101abe41578","Type":"ContainerDied","Data":"b023c095b83d75a867613cc48f045a4cfdb4c32105b95bb84c98dac2f4d0ac28"} Oct 30 05:58:14.937672 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.937654 5565 scope.go:122] "RemoveContainer" containerID="e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e" Oct 30 05:58:14.937931 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:58:14.937910 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e\": container with ID starting with e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e not found: ID does not exist" containerID="e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e" Oct 30 05:58:14.938015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.937936 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e"} err="failed to get container status \"e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e\": rpc error: code = NotFound desc = could not find container \"e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e\": container with ID starting with e13a16e434f158a36ba4e1dae338c4b08cdaa6956b0e711913875fc35690b37e not found: ID does not exist" Oct 30 05:58:14.938015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.937951 5565 scope.go:122] "RemoveContainer" containerID="a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6" Oct 30 05:58:14.948017 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.947999 5565 scope.go:122] "RemoveContainer" containerID="a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6" Oct 30 05:58:14.948266 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:58:14.948248 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6\": container with ID starting with a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6 not found: ID does not exist" containerID="a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6" Oct 30 05:58:14.948328 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:14.948271 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6"} err="failed to get container status \"a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6\": rpc error: code = NotFound desc = could not find container \"a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6\": container with ID starting with a22fece7322e152cec690d42c6629f1a263a7d74e75875767dcc4e4e45713db6 not found: ID does not exist" Oct 30 05:58:15.006621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:15.006588 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr"] Oct 30 05:58:15.012481 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:15.012454 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-a1c17ebc-9f9bd74cd-hdvpr"] Oct 30 05:58:15.050424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:15.050348 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4"] Oct 30 05:58:15.080165 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:15.080137 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-a1c17ebc-86f8d7f86c-ps2d4"] Oct 30 05:58:15.589697 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:15.589667 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a88a121e-ec20-4ec3-9a50-9101abe41578" path="/var/lib/kubelet/pods/a88a121e-ec20-4ec3-9a50-9101abe41578/volumes" Oct 30 05:58:15.590153 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:15.590137 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" path="/var/lib/kubelet/pods/cf41c9de-39ef-4393-8bb6-b57ea97ecfb5/volumes" Oct 30 05:58:17.540057 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:58:17.540017 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:58:18.992422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.992389 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws"] Oct 30 05:58:18.993333 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993315 5565 cpu_manager.go:414] "containerMap: removing container" podUID="53750c8f-13b4-4bdc-9d00-db931538c3af" containerName="revoke-keys" Oct 30 05:58:18.993381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993335 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53750c8f-13b4-4bdc-9d00-db931538c3af" containerName="revoke-keys" Oct 30 05:58:18.993381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993364 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" containerName="payload-pre-processing" Oct 30 05:58:18.993381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993369 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" containerName="payload-pre-processing" Oct 30 05:58:18.993381 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993378 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a88a121e-ec20-4ec3-9a50-9101abe41578" containerName="payload-processing" Oct 30 05:58:18.993510 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993382 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a88a121e-ec20-4ec3-9a50-9101abe41578" containerName="payload-processing" Oct 30 05:58:18.993578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993566 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="53750c8f-13b4-4bdc-9d00-db931538c3af" containerName="revoke-keys" Oct 30 05:58:18.993615 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993590 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf41c9de-39ef-4393-8bb6-b57ea97ecfb5" containerName="payload-pre-processing" Oct 30 05:58:18.993615 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:18.993601 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a88a121e-ec20-4ec3-9a50-9101abe41578" containerName="payload-processing" Oct 30 05:58:19.069455 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.069423 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws"] Oct 30 05:58:19.069661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.069563 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.086320 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.086289 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:58:19.086473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.086453 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:58:19.213585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.213545 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gh7z\" (UniqueName: \"kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.213585 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.213589 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.213837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.213620 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.315402 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.315362 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gh7z\" (UniqueName: \"kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.315641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.315412 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.315641 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.315463 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.316218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.316199 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.317918 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.317893 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.343365 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.343338 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gh7z\" (UniqueName: \"kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z\") pod \"maas-api-revoke-keys-e2e-api-b-530a5e-7qsws\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.450061 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.450032 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:19.893556 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:58:19.893506 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ded96ab_7f06_45b4_9453_253ac350281f.slice/crio-f598793af512b0d6bad7099cac7d59f9a49cee0868a7508bbaf20a160ec9a7ad WatchSource:0}: Error finding container f598793af512b0d6bad7099cac7d59f9a49cee0868a7508bbaf20a160ec9a7ad: Status 404 returned error can't find the container with id f598793af512b0d6bad7099cac7d59f9a49cee0868a7508bbaf20a160ec9a7ad Oct 30 05:58:19.895221 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:19.895155 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws"] Oct 30 05:58:20.082702 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:20.082665 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" event={"ID":"0ded96ab-7f06-45b4-9453-253ac350281f","Type":"ContainerStarted","Data":"f598793af512b0d6bad7099cac7d59f9a49cee0868a7508bbaf20a160ec9a7ad"} Oct 30 05:58:21.098692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:21.098609 5565 generic.go:356] "Generic (PLEG): container finished" podID="0ded96ab-7f06-45b4-9453-253ac350281f" containerID="578fa8ef80fd9b5663eb3090b011d61bd9a7e8c4624eae96be4b6285fc6f31d9" exitCode=0 Oct 30 05:58:21.099248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:21.099046 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" event={"ID":"0ded96ab-7f06-45b4-9453-253ac350281f","Type":"ContainerDied","Data":"578fa8ef80fd9b5663eb3090b011d61bd9a7e8c4624eae96be4b6285fc6f31d9"} Oct 30 05:58:22.113134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.113038 5565 generic.go:356] "Generic (PLEG): container finished" podID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerID="aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4" exitCode=0 Oct 30 05:58:22.113134 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.113116 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" event={"ID":"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0","Type":"ContainerDied","Data":"aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4"} Oct 30 05:58:22.114055 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.113754 5565 scope.go:122] "RemoveContainer" containerID="aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4" Oct 30 05:58:22.116461 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.116416 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:58:22.116642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.116556 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:58:22.116642 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.116608 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 05:58:22.259218 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.259183 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2"] Oct 30 05:58:22.271439 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.267846 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-df544d-2zhv2"] Oct 30 05:58:22.401610 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.395181 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:22.522590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.520289 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca\") pod \"0ded96ab-7f06-45b4-9453-253ac350281f\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " Oct 30 05:58:22.522590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.520918 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca\") pod \"0ded96ab-7f06-45b4-9453-253ac350281f\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " Oct 30 05:58:22.522590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.521363 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca" pod "0ded96ab-7f06-45b4-9453-253ac350281f" (UID: "0ded96ab-7f06-45b4-9453-253ac350281f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:22.522590 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.521435 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z\" (UniqueName: \"kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z\") pod \"0ded96ab-7f06-45b4-9453-253ac350281f\" (UID: \"0ded96ab-7f06-45b4-9453-253ac350281f\") " Oct 30 05:58:22.522940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.522733 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:22.529682 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.529613 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z" pod "0ded96ab-7f06-45b4-9453-253ac350281f" (UID: "0ded96ab-7f06-45b4-9453-253ac350281f"). InnerVolumeSpecName "kube-api-access-4gh7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:22.530128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.530092 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca" pod "0ded96ab-7f06-45b4-9453-253ac350281f" (UID: "0ded96ab-7f06-45b4-9453-253ac350281f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:22.625098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.625071 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ded96ab-7f06-45b4-9453-253ac350281f-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:22.625098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:22.625096 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gh7z\" (UniqueName: \"kubernetes.io/projected/0ded96ab-7f06-45b4-9453-253ac350281f-kube-api-access-4gh7z\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:23.035016 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.034981 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl"] Oct 30 05:58:23.049707 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.049675 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-dee2c484-zzqzl"] Oct 30 05:58:23.128714 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.128684 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" Oct 30 05:58:23.129186 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.128719 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws" event={"ID":"0ded96ab-7f06-45b4-9453-253ac350281f","Type":"ContainerDied","Data":"f598793af512b0d6bad7099cac7d59f9a49cee0868a7508bbaf20a160ec9a7ad"} Oct 30 05:58:23.129186 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.128766 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f598793af512b0d6bad7099cac7d59f9a49cee0868a7508bbaf20a160ec9a7ad" Oct 30 05:58:23.131184 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.131153 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" event={"ID":"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0","Type":"ContainerStarted","Data":"b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a"} Oct 30 05:58:23.587430 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.587396 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21571bc6-d4d6-40b7-a521-c91db6ec8cbd" path="/var/lib/kubelet/pods/21571bc6-d4d6-40b7-a521-c91db6ec8cbd/volumes" Oct 30 05:58:23.587991 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.587967 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b449b574-e46f-4d7c-9b76-8ff958a4f9fb" path="/var/lib/kubelet/pods/b449b574-e46f-4d7c-9b76-8ff958a4f9fb/volumes" Oct 30 05:58:23.757558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.753605 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:58:23.757558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.753724 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 05:58:23.757558 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:23.753765 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 05:58:24.758555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:24.756629 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 05:58:24.758555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:24.756694 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 05:58:25.187543 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:25.187443 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l"] Oct 30 05:58:25.192325 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:25.192295 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e94bd8-nw84l"] Oct 30 05:58:25.619387 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:25.618715 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b606f3c8-841d-4cf6-be9b-e323262fb52f" path="/var/lib/kubelet/pods/b606f3c8-841d-4cf6-be9b-e323262fb52f/volumes" Oct 30 05:58:25.782555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:25.761663 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 05:58:25.782555 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:25.761758 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 05:58:26.687683 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:26.687653 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:58:26.768561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:26.767843 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 05:58:26.768561 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:26.767899 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 05:58:27.754895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:27.754858 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 05:58:27.755473 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:27.755445 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 05:58:28.768728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:28.763617 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 05:58:28.768728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:28.763689 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 05:58:29.264518 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.264360 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8"] Oct 30 05:58:29.292858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.292831 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc"] Oct 30 05:58:29.697692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.697568 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp"] Oct 30 05:58:29.702803 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.702759 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" containerID="cri-o://332faf3279356de198e308f13c75ee70eaec591ebd4374ee63882ab8b8492f98" gracePeriod=30 Oct 30 05:58:29.708429 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.708398 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm"] Oct 30 05:58:29.708692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.708666 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" containerID="cri-o://1fffa6b1c0f8e0c69015fab7bd045db2187418c6c42ea5f5914053e98d9c9fea" gracePeriod=30 Oct 30 05:58:29.759383 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.759347 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:58:29.760382 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:29.760358 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 05:58:30.342382 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.342349 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7"] Oct 30 05:58:30.343605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.343579 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0ded96ab-7f06-45b4-9453-253ac350281f" containerName="revoke-keys" Oct 30 05:58:30.343605 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.343602 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ded96ab-7f06-45b4-9453-253ac350281f" containerName="revoke-keys" Oct 30 05:58:30.343865 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.343849 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ded96ab-7f06-45b4-9453-253ac350281f" containerName="revoke-keys" Oct 30 05:58:30.348739 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.348710 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.364311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.364282 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5mkz\" (UniqueName: \"kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.364595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.364574 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.365015 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.364993 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.365858 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.365834 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7"] Oct 30 05:58:30.381917 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.381896 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:58:30.382005 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.381991 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:58:30.466148 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.466120 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.466309 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.466294 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.466418 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.466403 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5mkz\" (UniqueName: \"kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.467004 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.466977 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.468437 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.468415 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.491935 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.491913 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5mkz\" (UniqueName: \"kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz\") pod \"maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.666772 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.666700 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:30.981166 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:30.981134 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7"] Oct 30 05:58:30.981811 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:58:30.981784 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfec559fb_fc7a_4493_aadd_643b6a57e6ba.slice/crio-10d8bd03480b03b8cde8ebc44c65d836ed922c18db50cfd5870fe0447a3813a6 WatchSource:0}: Error finding container 10d8bd03480b03b8cde8ebc44c65d836ed922c18db50cfd5870fe0447a3813a6: Status 404 returned error can't find the container with id 10d8bd03480b03b8cde8ebc44c65d836ed922c18db50cfd5870fe0447a3813a6 Oct 30 05:58:31.234179 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:31.234099 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" event={"ID":"fec559fb-fc7a-4493-aadd-643b6a57e6ba","Type":"ContainerStarted","Data":"10d8bd03480b03b8cde8ebc44c65d836ed922c18db50cfd5870fe0447a3813a6"} Oct 30 05:58:32.247721 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:32.247641 5565 generic.go:356] "Generic (PLEG): container finished" podID="fec559fb-fc7a-4493-aadd-643b6a57e6ba" containerID="f1e6521f2ca93ecbd4f667867a49658a6779e65a998bee0179073618b81af730" exitCode=0 Oct 30 05:58:32.248096 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:32.247737 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" event={"ID":"fec559fb-fc7a-4493-aadd-643b6a57e6ba","Type":"ContainerDied","Data":"f1e6521f2ca93ecbd4f667867a49658a6779e65a998bee0179073618b81af730"} Oct 30 05:58:33.447124 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.447092 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:33.535208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.535138 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca\") pod \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " Oct 30 05:58:33.535358 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.535291 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca\") pod \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " Oct 30 05:58:33.535563 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.535509 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz\" (UniqueName: \"kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz\") pod \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\" (UID: \"fec559fb-fc7a-4493-aadd-643b6a57e6ba\") " Oct 30 05:58:33.536032 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.536005 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca" pod "fec559fb-fc7a-4493-aadd-643b6a57e6ba" (UID: "fec559fb-fc7a-4493-aadd-643b6a57e6ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:33.536549 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.536505 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:33.537883 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.537838 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca" pod "fec559fb-fc7a-4493-aadd-643b6a57e6ba" (UID: "fec559fb-fc7a-4493-aadd-643b6a57e6ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:33.538792 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.538766 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz" pod "fec559fb-fc7a-4493-aadd-643b6a57e6ba" (UID: "fec559fb-fc7a-4493-aadd-643b6a57e6ba"). InnerVolumeSpecName "kube-api-access-b5mkz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:33.638732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.638703 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5mkz\" (UniqueName: \"kubernetes.io/projected/fec559fb-fc7a-4493-aadd-643b6a57e6ba-kube-api-access-b5mkz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:33.638732 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:33.638735 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fec559fb-fc7a-4493-aadd-643b6a57e6ba-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:34.176827 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.176793 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc"] Oct 30 05:58:34.177606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.177589 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fec559fb-fc7a-4493-aadd-643b6a57e6ba" containerName="revoke-keys" Oct 30 05:58:34.177606 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.177605 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fec559fb-fc7a-4493-aadd-643b6a57e6ba" containerName="revoke-keys" Oct 30 05:58:34.177771 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.177760 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fec559fb-fc7a-4493-aadd-643b6a57e6ba" containerName="revoke-keys" Oct 30 05:58:34.181658 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.181639 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.210727 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.209922 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc"] Oct 30 05:58:34.257635 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.257600 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.257814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.257671 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vzms\" (UniqueName: \"kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.257814 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.257765 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.265293 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.265260 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" podUID="5695cff4-f0dd-4f1a-9a44-86e614677b48" containerName="payload-processing" containerID="cri-o://94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4" gracePeriod=25 Oct 30 05:58:34.274750 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.274719 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" event={"ID":"fec559fb-fc7a-4493-aadd-643b6a57e6ba","Type":"ContainerDied","Data":"10d8bd03480b03b8cde8ebc44c65d836ed922c18db50cfd5870fe0447a3813a6"} Oct 30 05:58:34.274886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.274778 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="10d8bd03480b03b8cde8ebc44c65d836ed922c18db50cfd5870fe0447a3813a6" Oct 30 05:58:34.274886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.274841 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7" Oct 30 05:58:34.295481 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.294658 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" podUID="6ff3ac10-bdb7-4868-9538-04113c5b677e" containerName="payload-pre-processing" containerID="cri-o://1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3" gracePeriod=25 Oct 30 05:58:34.359692 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.359666 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vzms\" (UniqueName: \"kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.359816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.359801 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.360145 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.360126 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.360758 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.360735 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.362185 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.362159 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.387775 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.387748 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vzms\" (UniqueName: \"kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms\") pod \"maas-api-revoke-keys-e2e-api-b-76b905-9m6mc\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:34.496695 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:34.496403 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:35.129675 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.128478 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:58:35.292548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.277282 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume\") pod \"5695cff4-f0dd-4f1a-9a44-86e614677b48\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " Oct 30 05:58:35.293889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.293859 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md\" (UniqueName: \"kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md\") pod \"5695cff4-f0dd-4f1a-9a44-86e614677b48\" (UID: \"5695cff4-f0dd-4f1a-9a44-86e614677b48\") " Oct 30 05:58:35.351603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.331136 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:58:35.375577 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.375542 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md" pod "5695cff4-f0dd-4f1a-9a44-86e614677b48" (UID: "5695cff4-f0dd-4f1a-9a44-86e614677b48"). InnerVolumeSpecName "kube-api-access-872md". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.378094 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.375384 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume" pod "5695cff4-f0dd-4f1a-9a44-86e614677b48" (UID: "5695cff4-f0dd-4f1a-9a44-86e614677b48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:35.435943 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.435919 5565 generic.go:356] "Generic (PLEG): container finished" podID="6ff3ac10-bdb7-4868-9538-04113c5b677e" containerID="1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3" exitCode=0 Oct 30 05:58:35.436156 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.436111 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" event={"ID":"6ff3ac10-bdb7-4868-9538-04113c5b677e","Type":"ContainerDied","Data":"1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3"} Oct 30 05:58:35.436301 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.436289 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" event={"ID":"6ff3ac10-bdb7-4868-9538-04113c5b677e","Type":"ContainerDied","Data":"c9abbc8497ab358a82e3e243ffbca9739e2c6a8356541733a8ded5db641dd46c"} Oct 30 05:58:35.436388 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.436379 5565 scope.go:122] "RemoveContainer" containerID="1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3" Oct 30 05:58:35.436630 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.436612 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc" Oct 30 05:58:35.443458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.440504 5565 generic.go:356] "Generic (PLEG): container finished" podID="5695cff4-f0dd-4f1a-9a44-86e614677b48" containerID="94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4" exitCode=0 Oct 30 05:58:35.443458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.440736 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" event={"ID":"5695cff4-f0dd-4f1a-9a44-86e614677b48","Type":"ContainerDied","Data":"94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4"} Oct 30 05:58:35.443458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.440779 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" event={"ID":"5695cff4-f0dd-4f1a-9a44-86e614677b48","Type":"ContainerDied","Data":"4dc556fbab7ae97b11bf4a55e3e452bbb766e209360079a22de9902bcf7d048e"} Oct 30 05:58:35.443458 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.440858 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8" Oct 30 05:58:35.451169 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.451125 5565 generic.go:356] "Generic (PLEG): container finished" podID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerID="332faf3279356de198e308f13c75ee70eaec591ebd4374ee63882ab8b8492f98" exitCode=0 Oct 30 05:58:35.451377 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.451322 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" event={"ID":"943f3183-3c07-464e-980d-69b02e7a1a8f","Type":"ContainerDied","Data":"332faf3279356de198e308f13c75ee70eaec591ebd4374ee63882ab8b8492f98"} Oct 30 05:58:35.457427 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.457242 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn\" (UniqueName: \"kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn\") pod \"6ff3ac10-bdb7-4868-9538-04113c5b677e\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " Oct 30 05:58:35.457500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.457469 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume\") pod \"6ff3ac10-bdb7-4868-9538-04113c5b677e\" (UID: \"6ff3ac10-bdb7-4868-9538-04113c5b677e\") " Oct 30 05:58:35.457925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.457879 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume" pod "6ff3ac10-bdb7-4868-9538-04113c5b677e" (UID: "6ff3ac10-bdb7-4868-9538-04113c5b677e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:35.459261 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.459236 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-872md\" (UniqueName: \"kubernetes.io/projected/5695cff4-f0dd-4f1a-9a44-86e614677b48-kube-api-access-872md\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.459398 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.459385 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff3ac10-bdb7-4868-9538-04113c5b677e-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.459491 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.459480 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5695cff4-f0dd-4f1a-9a44-86e614677b48-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.461925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.461829 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn" pod "6ff3ac10-bdb7-4868-9538-04113c5b677e" (UID: "6ff3ac10-bdb7-4868-9538-04113c5b677e"). InnerVolumeSpecName "kube-api-access-wjkbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.462196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.462098 5565 generic.go:356] "Generic (PLEG): container finished" podID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerID="1fffa6b1c0f8e0c69015fab7bd045db2187418c6c42ea5f5914053e98d9c9fea" exitCode=0 Oct 30 05:58:35.462196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.462148 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" event={"ID":"2748ccfa-1021-40ac-9d37-ba329b658eba","Type":"ContainerDied","Data":"1fffa6b1c0f8e0c69015fab7bd045db2187418c6c42ea5f5914053e98d9c9fea"} Oct 30 05:58:35.462608 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.462592 5565 scope.go:122] "RemoveContainer" containerID="1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3" Oct 30 05:58:35.463177 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:58:35.463151 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3\": container with ID starting with 1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3 not found: ID does not exist" containerID="1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3" Oct 30 05:58:35.463380 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.463356 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3"} err="failed to get container status \"1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3\": rpc error: code = NotFound desc = could not find container \"1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3\": container with ID starting with 1b146b16306f5d8e3e25e6fa4fb1ce122f7ccc03ff9aa99ac16be073bbe582f3 not found: ID does not exist" Oct 30 05:58:35.463604 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.463504 5565 scope.go:122] "RemoveContainer" containerID="94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4" Oct 30 05:58:35.511696 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.511643 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc"] Oct 30 05:58:35.539959 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.539032 5565 scope.go:122] "RemoveContainer" containerID="94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4" Oct 30 05:58:35.555324 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:58:35.543483 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4\": container with ID starting with 94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4 not found: ID does not exist" containerID="94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4" Oct 30 05:58:35.555324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.543612 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4"} err="failed to get container status \"94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4\": rpc error: code = NotFound desc = could not find container \"94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4\": container with ID starting with 94f4bc3ba80a357f8ead065053cdba07c6b7a89b50ce987af944339ed7dcebd4 not found: ID does not exist" Oct 30 05:58:35.555324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.546366 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc"] Oct 30 05:58:35.555324 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.554266 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-a1c17ebc-f2wsc"] Oct 30 05:58:35.571895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.569733 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjkbn\" (UniqueName: \"kubernetes.io/projected/6ff3ac10-bdb7-4868-9538-04113c5b677e-kube-api-access-wjkbn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.587595 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.586829 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53750c8f-13b4-4bdc-9d00-db931538c3af" path="/var/lib/kubelet/pods/53750c8f-13b4-4bdc-9d00-db931538c3af/volumes" Oct 30 05:58:35.587774 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.587754 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8"] Oct 30 05:58:35.587856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.587796 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-530a5e-94d87598-krxs8"] Oct 30 05:58:35.660546 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.647810 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775308 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775369 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775521 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775573 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775642 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775722 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.775849 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.776061 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.776140 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy\") pod \"2748ccfa-1021-40ac-9d37-ba329b658eba\" (UID: \"2748ccfa-1021-40ac-9d37-ba329b658eba\") " Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.778413 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.778756 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.779000 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.779178 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.779681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.779347 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.802951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.802778 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc"] Oct 30 05:58:35.816795 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.816279 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.817110 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.817063 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:35.819699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.819675 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-530a5e-6f4cf79b47-hnnwc"] Oct 30 05:58:35.821298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.821271 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.821821 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.821795 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c" pod "2748ccfa-1021-40ac-9d37-ba329b658eba" (UID: "2748ccfa-1021-40ac-9d37-ba329b658eba"). InnerVolumeSpecName "kube-api-access-9fb7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.829023 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.829004 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:58:35.878746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878719 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878746 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878746 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878762 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878771 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2748ccfa-1021-40ac-9d37-ba329b658eba-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878780 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fb7c\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-kube-api-access-9fb7c\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878790 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878798 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878806 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.878925 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.878814 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2748ccfa-1021-40ac-9d37-ba329b658eba-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.979452 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979429 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.979623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979491 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.979623 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979586 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.979749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979634 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.979749 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979690 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.979857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979806 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.979857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979827 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.979857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979839 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.980057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979881 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.980057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979961 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.980057 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.979977 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.980211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980063 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:35.980211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980161 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo\") pod \"943f3183-3c07-464e-980d-69b02e7a1a8f\" (UID: \"943f3183-3c07-464e-980d-69b02e7a1a8f\") " Oct 30 05:58:35.980311 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980263 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.980952 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980932 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.981060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980954 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.981060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980970 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/943f3183-3c07-464e-980d-69b02e7a1a8f-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.981060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980982 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.981060 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.980994 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:35.982347 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.982326 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:58:35.982503 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.982482 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:58:35.982614 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.982591 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:35.982715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:35.982696 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp" pod "943f3183-3c07-464e-980d-69b02e7a1a8f" (UID: "943f3183-3c07-464e-980d-69b02e7a1a8f"). InnerVolumeSpecName "kube-api-access-6svmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:36.082854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.082789 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.082854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.082813 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6svmp\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-kube-api-access-6svmp\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.082854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.082824 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.082854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.082832 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/943f3183-3c07-464e-980d-69b02e7a1a8f-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:36.473395 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.473369 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" Oct 30 05:58:36.473588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.473371 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp" event={"ID":"943f3183-3c07-464e-980d-69b02e7a1a8f","Type":"ContainerDied","Data":"51c50b418129e771345506af94cae98ee7fee2b598bf4a9fd19d45c4177b600c"} Oct 30 05:58:36.473588 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.473506 5565 scope.go:122] "RemoveContainer" containerID="332faf3279356de198e308f13c75ee70eaec591ebd4374ee63882ab8b8492f98" Oct 30 05:58:36.475174 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.475148 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" event={"ID":"e4baa3f0-f9da-4c06-a756-a0bae00da117","Type":"ContainerStarted","Data":"0a6015da1b44922a8984faf5a2a707f201217d5d81ed2aac1ce26228cf6e60b6"} Oct 30 05:58:36.475306 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.475195 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" event={"ID":"e4baa3f0-f9da-4c06-a756-a0bae00da117","Type":"ContainerStarted","Data":"b5931a3c20c26711230e3af69aae7a26686ccfcd2564095e0dbe8e33d1d4d787"} Oct 30 05:58:36.476731 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.476708 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" event={"ID":"2748ccfa-1021-40ac-9d37-ba329b658eba","Type":"ContainerDied","Data":"7c2f2b8fc2feae5b2d1f0a939a6c5e071d918f78c365c8ea65ffbd60beecfa90"} Oct 30 05:58:36.476846 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.476761 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm" Oct 30 05:58:36.487603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.487432 5565 scope.go:122] "RemoveContainer" containerID="1fffa6b1c0f8e0c69015fab7bd045db2187418c6c42ea5f5914053e98d9c9fea" Oct 30 05:58:36.538860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.538828 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp"] Oct 30 05:58:36.552936 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.552901 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-d7jqp"] Oct 30 05:58:36.585468 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.585425 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" podStartSLOduration=2.585411577 podStartE2EDuration="2.585411577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 05:58:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 05:58:36.583480572 +0000 UTC m=+1623.159167630" watchObservedRunningTime="2026-10-30 05:58:36.585411577 +0000 UTC m=+1623.161098631" Oct 30 05:58:36.626715 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.626690 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm"] Oct 30 05:58:36.631446 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:36.631425 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-df544d-openshift-default-bdd854567-8wbgm"] Oct 30 05:58:37.182123 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.182094 5565 scope.go:122] "RemoveContainer" containerID="be46412fad008b7198ff0823d55add2557c5f20e51dd3fb913317f9212a72370" Oct 30 05:58:37.194356 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.194332 5565 scope.go:122] "RemoveContainer" containerID="9ac06da60fb9945f049e2f445e5e726e6ff290f320b67930a758546da596651d" Oct 30 05:58:37.205128 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.205106 5565 scope.go:122] "RemoveContainer" containerID="832eacf9cc2642f6712dd9992de3ba6c173cfee40b8fd822a9f3bca8c0051e02" Oct 30 05:58:37.552281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.552235 5565 generic.go:356] "Generic (PLEG): container finished" podID="e4baa3f0-f9da-4c06-a756-a0bae00da117" containerID="0a6015da1b44922a8984faf5a2a707f201217d5d81ed2aac1ce26228cf6e60b6" exitCode=0 Oct 30 05:58:37.553034 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.552998 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" event={"ID":"e4baa3f0-f9da-4c06-a756-a0bae00da117","Type":"ContainerDied","Data":"0a6015da1b44922a8984faf5a2a707f201217d5d81ed2aac1ce26228cf6e60b6"} Oct 30 05:58:37.612101 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.612060 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" path="/var/lib/kubelet/pods/2748ccfa-1021-40ac-9d37-ba329b658eba/volumes" Oct 30 05:58:37.612913 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.612884 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5695cff4-f0dd-4f1a-9a44-86e614677b48" path="/var/lib/kubelet/pods/5695cff4-f0dd-4f1a-9a44-86e614677b48/volumes" Oct 30 05:58:37.613572 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.613550 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ff3ac10-bdb7-4868-9538-04113c5b677e" path="/var/lib/kubelet/pods/6ff3ac10-bdb7-4868-9538-04113c5b677e/volumes" Oct 30 05:58:37.614421 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:37.614403 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" path="/var/lib/kubelet/pods/943f3183-3c07-464e-980d-69b02e7a1a8f/volumes" Oct 30 05:58:38.759681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.759652 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:38.848490 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.848403 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms\" (UniqueName: \"kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms\") pod \"e4baa3f0-f9da-4c06-a756-a0bae00da117\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " Oct 30 05:58:38.848713 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.848685 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca\") pod \"e4baa3f0-f9da-4c06-a756-a0bae00da117\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " Oct 30 05:58:38.848860 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.848797 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca\") pod \"e4baa3f0-f9da-4c06-a756-a0bae00da117\" (UID: \"e4baa3f0-f9da-4c06-a756-a0bae00da117\") " Oct 30 05:58:38.849181 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.849153 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca" pod "e4baa3f0-f9da-4c06-a756-a0bae00da117" (UID: "e4baa3f0-f9da-4c06-a756-a0bae00da117"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:58:38.849730 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.849710 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:38.854579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.851183 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms" pod "e4baa3f0-f9da-4c06-a756-a0bae00da117" (UID: "e4baa3f0-f9da-4c06-a756-a0bae00da117"). InnerVolumeSpecName "kube-api-access-7vzms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:58:38.854579 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.851286 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca" pod "e4baa3f0-f9da-4c06-a756-a0bae00da117" (UID: "e4baa3f0-f9da-4c06-a756-a0bae00da117"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:58:38.951557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.951511 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vzms\" (UniqueName: \"kubernetes.io/projected/e4baa3f0-f9da-4c06-a756-a0bae00da117-kube-api-access-7vzms\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:38.951557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:38.951556 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4baa3f0-f9da-4c06-a756-a0bae00da117-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:58:39.571798 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:39.571765 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" Oct 30 05:58:39.573092 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:39.573062 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc" event={"ID":"e4baa3f0-f9da-4c06-a756-a0bae00da117","Type":"ContainerDied","Data":"b5931a3c20c26711230e3af69aae7a26686ccfcd2564095e0dbe8e33d1d4d787"} Oct 30 05:58:39.573211 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:39.573118 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5931a3c20c26711230e3af69aae7a26686ccfcd2564095e0dbe8e33d1d4d787" Oct 30 05:58:47.536506 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:58:47.536413 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:58:53.047296 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:53.047226 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s"] Oct 30 05:58:53.053923 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:53.053886 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-15e2832d-dcq8s"] Oct 30 05:58:53.569635 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:58:53.569604 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c935779e-20ad-42e6-982f-74ca32d9ded2" path="/var/lib/kubelet/pods/c935779e-20ad-42e6-982f-74ca32d9ded2/volumes" Oct 30 05:59:01.187463 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:01.187424 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 05:59:04.803084 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:04.803041 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm"] Oct 30 05:59:04.803578 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:04.803253 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" containerID="cri-o://12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e" gracePeriod=30 Oct 30 05:59:10.119634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.119605 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:59:10.224224 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224190 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224275 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224303 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224328 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224366 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224421 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224500 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224698 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224688 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224748 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo\") pod \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\" (UID: \"1ddb6c6b-95c6-4022-a4b6-0228a827cf85\") " Oct 30 05:59:10.224874 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224679 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.224982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224917 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.224982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224934 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.224982 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.224955 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.225126 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.225101 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:10.225469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.225448 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.225469 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.225471 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.225737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.225486 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.225737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.225497 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.225737 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.225511 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.227171 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.227143 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "kube-api-access-jgccz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:10.227279 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.227197 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:10.227366 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.227345 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:10.227634 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.227614 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token" pod "1ddb6c6b-95c6-4022-a4b6-0228a827cf85" (UID: "1ddb6c6b-95c6-4022-a4b6-0228a827cf85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:10.327248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.327217 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgccz\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-kube-api-access-jgccz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.327248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.327246 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.327248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.327256 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:10.327248 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:10.327265 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1ddb6c6b-95c6-4022-a4b6-0228a827cf85-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:11.084728 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.084510 5565 generic.go:356] "Generic (PLEG): container finished" podID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerID="12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e" exitCode=0 Oct 30 05:59:11.085095 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.085063 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" event={"ID":"1ddb6c6b-95c6-4022-a4b6-0228a827cf85","Type":"ContainerDied","Data":"12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e"} Oct 30 05:59:11.085326 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.085295 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" event={"ID":"1ddb6c6b-95c6-4022-a4b6-0228a827cf85","Type":"ContainerDied","Data":"4795d8aa7802cacf14041659cf512ff860e5e47e5dbeffc5363c886bc9e50d9f"} Oct 30 05:59:11.085488 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.085472 5565 scope.go:122] "RemoveContainer" containerID="12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e" Oct 30 05:59:11.085854 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.085837 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm" Oct 30 05:59:11.115741 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.115707 5565 scope.go:122] "RemoveContainer" containerID="12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e" Oct 30 05:59:11.116080 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:59:11.116056 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e\": container with ID starting with 12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e not found: ID does not exist" containerID="12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e" Oct 30 05:59:11.116178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.116092 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e"} err="failed to get container status \"12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e\": rpc error: code = NotFound desc = could not find container \"12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e\": container with ID starting with 12e5bca76b06e969d5735634bdac89d4677725bb84f9ad39c0092c129579748e not found: ID does not exist" Oct 30 05:59:11.165348 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.165319 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm"] Oct 30 05:59:11.169807 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.169786 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-8h5wm"] Oct 30 05:59:11.569436 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:11.569405 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" path="/var/lib/kubelet/pods/1ddb6c6b-95c6-4022-a4b6-0228a827cf85/volumes" Oct 30 05:59:17.534843 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:59:17.534802 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 05:59:29.580208 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580171 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp"] Oct 30 05:59:29.580945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580929 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6ff3ac10-bdb7-4868-9538-04113c5b677e" containerName="payload-pre-processing" Oct 30 05:59:29.580997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580946 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ff3ac10-bdb7-4868-9538-04113c5b677e" containerName="payload-pre-processing" Oct 30 05:59:29.580997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580963 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5695cff4-f0dd-4f1a-9a44-86e614677b48" containerName="payload-processing" Oct 30 05:59:29.580997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580968 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5695cff4-f0dd-4f1a-9a44-86e614677b48" containerName="payload-processing" Oct 30 05:59:29.580997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580990 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" Oct 30 05:59:29.580997 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.580995 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581005 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e4baa3f0-f9da-4c06-a756-a0bae00da117" containerName="revoke-keys" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581010 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4baa3f0-f9da-4c06-a756-a0bae00da117" containerName="revoke-keys" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581020 5565 cpu_manager.go:414] "containerMap: removing container" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581025 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581046 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581051 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" Oct 30 05:59:29.581163 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581163 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ff3ac10-bdb7-4868-9538-04113c5b677e" containerName="payload-pre-processing" Oct 30 05:59:29.581424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581173 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5695cff4-f0dd-4f1a-9a44-86e614677b48" containerName="payload-processing" Oct 30 05:59:29.581424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581183 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddb6c6b-95c6-4022-a4b6-0228a827cf85" containerName="istio-proxy" Oct 30 05:59:29.581424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581190 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4baa3f0-f9da-4c06-a756-a0bae00da117" containerName="revoke-keys" Oct 30 05:59:29.581424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581197 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="943f3183-3c07-464e-980d-69b02e7a1a8f" containerName="istio-proxy" Oct 30 05:59:29.581424 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.581205 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2748ccfa-1021-40ac-9d37-ba329b658eba" containerName="istio-proxy" Oct 30 05:59:29.585000 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.584981 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.625718 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.625692 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 05:59:29.625885 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.625724 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 05:59:29.631042 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.631015 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp"] Oct 30 05:59:29.666557 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.666495 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8tfk\" (UniqueName: \"kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.666929 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.666905 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.667046 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.666985 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.769249 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.769220 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.769411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.769270 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.769411 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.769341 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8tfk\" (UniqueName: \"kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.769957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.769928 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.771621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.771602 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.796603 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.796574 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8tfk\" (UniqueName: \"kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk\") pod \"maas-api-revoke-sub-0268343a-8t6sp\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:29.899699 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:29.899639 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:30.209024 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:30.208994 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp"] Oct 30 05:59:30.209795 ip-10-0-129-175 kubenswrapper[5565]: W1030 05:59:30.209766 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20edd61a_214b_42bd_a4cb_df979476fa73.slice/crio-0d4aabef40f791291393aa24a8e1f34b07bec8558766f3d407d725e17461246d WatchSource:0}: Error finding container 0d4aabef40f791291393aa24a8e1f34b07bec8558766f3d407d725e17461246d: Status 404 returned error can't find the container with id 0d4aabef40f791291393aa24a8e1f34b07bec8558766f3d407d725e17461246d Oct 30 05:59:30.278516 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:30.278485 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" event={"ID":"20edd61a-214b-42bd-a4cb-df979476fa73","Type":"ContainerStarted","Data":"0d4aabef40f791291393aa24a8e1f34b07bec8558766f3d407d725e17461246d"} Oct 30 05:59:30.447231 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:30.447203 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf"] Oct 30 05:59:30.447408 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:30.447388 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" containerID="cri-o://77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03" gracePeriod=30 Oct 30 05:59:31.287762 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:31.287732 5565 generic.go:356] "Generic (PLEG): container finished" podID="20edd61a-214b-42bd-a4cb-df979476fa73" containerID="243f5b3ef2c0628913403f5a86d8ae751ea467794382040f52336ad772908c3d" exitCode=0 Oct 30 05:59:31.288178 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:31.287823 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" event={"ID":"20edd61a-214b-42bd-a4cb-df979476fa73","Type":"ContainerDied","Data":"243f5b3ef2c0628913403f5a86d8ae751ea467794382040f52336ad772908c3d"} Oct 30 05:59:32.587951 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.565922 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:32.767738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.767700 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca\") pod \"20edd61a-214b-42bd-a4cb-df979476fa73\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " Oct 30 05:59:32.771281 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.771245 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk\" (UniqueName: \"kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk\") pod \"20edd61a-214b-42bd-a4cb-df979476fa73\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " Oct 30 05:59:32.771500 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.771475 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca\") pod \"20edd61a-214b-42bd-a4cb-df979476fa73\" (UID: \"20edd61a-214b-42bd-a4cb-df979476fa73\") " Oct 30 05:59:32.772895 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.772867 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca" pod "20edd61a-214b-42bd-a4cb-df979476fa73" (UID: "20edd61a-214b-42bd-a4cb-df979476fa73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:32.791308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.782699 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca" pod "20edd61a-214b-42bd-a4cb-df979476fa73" (UID: "20edd61a-214b-42bd-a4cb-df979476fa73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 05:59:32.791308 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.782826 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk" pod "20edd61a-214b-42bd-a4cb-df979476fa73" (UID: "20edd61a-214b-42bd-a4cb-df979476fa73"). InnerVolumeSpecName "kube-api-access-w8tfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:32.874414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.874311 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w8tfk\" (UniqueName: \"kubernetes.io/projected/20edd61a-214b-42bd-a4cb-df979476fa73-kube-api-access-w8tfk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:32.874414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.874348 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:32.874414 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:32.874361 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/20edd61a-214b-42bd-a4cb-df979476fa73-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:33.260887 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:33.260853 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws"] Oct 30 05:59:33.307138 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:33.295788 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-530a5e-7qsws"] Oct 30 05:59:33.319008 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:33.318954 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" Oct 30 05:59:33.319204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:33.318979 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp" event={"ID":"20edd61a-214b-42bd-a4cb-df979476fa73","Type":"ContainerDied","Data":"0d4aabef40f791291393aa24a8e1f34b07bec8558766f3d407d725e17461246d"} Oct 30 05:59:33.319204 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:33.319150 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d4aabef40f791291393aa24a8e1f34b07bec8558766f3d407d725e17461246d" Oct 30 05:59:33.570346 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:33.570275 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ded96ab-7f06-45b4-9453-253ac350281f" path="/var/lib/kubelet/pods/0ded96ab-7f06-45b4-9453-253ac350281f/volumes" Oct 30 05:59:34.254886 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:34.254854 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2"] Oct 30 05:59:34.255283 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:34.255039 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" containerID="cri-o://b2aee50c4fe9897573e4b077c60c0cb444f03dd8d02653c56598785cdac01a92" gracePeriod=30 Oct 30 05:59:34.548442 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:34.548365 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd"] Oct 30 05:59:34.548620 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:34.548581 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" containerID="cri-o://2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e" gracePeriod=30 Oct 30 05:59:35.736939 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.736882 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:59:35.827624 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.827595 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.827779 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.827726 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.827831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.827784 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.827831 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.827824 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.827957 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.827940 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.828196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828025 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.828196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828065 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.828196 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828094 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:35.828413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828226 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.828413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828206 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:35.828413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828263 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data\") pod \"40f54885-68d7-439b-b786-d314d7af6eb0\" (UID: \"40f54885-68d7-439b-b786-d314d7af6eb0\") " Oct 30 05:59:35.828413 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828400 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:35.828666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828427 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:35.828666 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828654 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:35.829065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828969 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.829065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.828996 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.829065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.829011 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.829065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.829026 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.829065 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.829038 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40f54885-68d7-439b-b786-d314d7af6eb0-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.830616 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.830578 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "kube-api-access-wqqq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:35.830764 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.830738 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:35.831286 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.831250 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:35.831449 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.831427 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy" pod "40f54885-68d7-439b-b786-d314d7af6eb0" (UID: "40f54885-68d7-439b-b786-d314d7af6eb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:35.930432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.930397 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40f54885-68d7-439b-b786-d314d7af6eb0-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.930432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.930428 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.930432 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.930438 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqq9\" (UniqueName: \"kubernetes.io/projected/40f54885-68d7-439b-b786-d314d7af6eb0-kube-api-access-wqqq9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:35.930690 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:35.930448 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40f54885-68d7-439b-b786-d314d7af6eb0-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:36.347511 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.347470 5565 generic.go:356] "Generic (PLEG): container finished" podID="40f54885-68d7-439b-b786-d314d7af6eb0" containerID="77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03" exitCode=0 Oct 30 05:59:36.347761 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.347736 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" event={"ID":"40f54885-68d7-439b-b786-d314d7af6eb0","Type":"ContainerDied","Data":"77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03"} Oct 30 05:59:36.347856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.347798 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" event={"ID":"40f54885-68d7-439b-b786-d314d7af6eb0","Type":"ContainerDied","Data":"a39cb8066f4cf2884b25898d31383abd08b56deb75b5b1a174c0ab1cdaeaf7a1"} Oct 30 05:59:36.347856 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.347829 5565 scope.go:122] "RemoveContainer" containerID="77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03" Oct 30 05:59:36.348006 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.347986 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf" Oct 30 05:59:36.362013 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.361988 5565 scope.go:122] "RemoveContainer" containerID="77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03" Oct 30 05:59:36.362370 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:59:36.362340 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03\": container with ID starting with 77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03 not found: ID does not exist" containerID="77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03" Oct 30 05:59:36.362470 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.362376 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03"} err="failed to get container status \"77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03\": rpc error: code = NotFound desc = could not find container \"77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03\": container with ID starting with 77be285065ff9401325a0ca96f75a077925167d68771792e733ad3999c68de03 not found: ID does not exist" Oct 30 05:59:36.461483 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.461450 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf"] Oct 30 05:59:36.473929 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:36.473899 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-b44bf"] Oct 30 05:59:37.379139 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:37.379101 5565 scope.go:122] "RemoveContainer" containerID="53ac9595bd1dc713bd980a0efae0045e63f5d26ffec690a2b295691be2d16bf6" Oct 30 05:59:37.570650 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:37.570617 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" path="/var/lib/kubelet/pods/40f54885-68d7-439b-b786-d314d7af6eb0/volumes" Oct 30 05:59:39.315209 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.315183 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7"] Oct 30 05:59:39.330837 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.329851 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-4323f6-ptpt7"] Oct 30 05:59:39.376664 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.376628 5565 generic.go:356] "Generic (PLEG): container finished" podID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerID="b2aee50c4fe9897573e4b077c60c0cb444f03dd8d02653c56598785cdac01a92" exitCode=0 Oct 30 05:59:39.376938 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.376774 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" event={"ID":"140e5755-d9a3-428f-8bd2-616cabc92c2a","Type":"ContainerDied","Data":"b2aee50c4fe9897573e4b077c60c0cb444f03dd8d02653c56598785cdac01a92"} Oct 30 05:59:39.571291 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.571265 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fec559fb-fc7a-4493-aadd-643b6a57e6ba" path="/var/lib/kubelet/pods/fec559fb-fc7a-4493-aadd-643b6a57e6ba/volumes" Oct 30 05:59:39.655298 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.655263 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:59:39.696722 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.696691 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.696857 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.696826 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.696926 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.696875 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.696983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.696952 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.696983 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.696959 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.697096 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.697023 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.697300 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.697273 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.697417 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.697322 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.698940 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.698915 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "kube-api-access-8qftj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:39.699047 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.698963 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:39.699638 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.699621 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.699686 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.699666 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.699738 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.699728 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.699823 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.699812 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo\") pod \"140e5755-d9a3-428f-8bd2-616cabc92c2a\" (UID: \"140e5755-d9a3-428f-8bd2-616cabc92c2a\") " Oct 30 05:59:39.699916 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.699894 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.700137 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700113 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700775 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700801 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qftj\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-kube-api-access-8qftj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700814 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700828 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700841 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700854 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.700889 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.700867 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/140e5755-d9a3-428f-8bd2-616cabc92c2a-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.701677 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.701655 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:39.703009 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.702987 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy" pod "140e5755-d9a3-428f-8bd2-616cabc92c2a" (UID: "140e5755-d9a3-428f-8bd2-616cabc92c2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.802869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.802783 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.802869 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.802815 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/140e5755-d9a3-428f-8bd2-616cabc92c2a-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.850777 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.850754 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:59:39.903407 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903348 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.903599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903445 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.903599 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903481 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.903729 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903705 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.904422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903747 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.904422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903825 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.904422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903874 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.904422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.903922 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.904422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.904011 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket\") pod \"37723510-9540-4794-b253-9e85fc0f6846\" (UID: \"37723510-9540-4794-b253-9e85fc0f6846\") " Oct 30 05:59:39.904422 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.904155 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.904816 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.904795 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:39.905033 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.905012 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.905229 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.905197 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.905742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.905675 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.905742 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.905592 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 05:59:39.906905 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.906881 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 05:59:39.909867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.909820 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 05:59:39.909867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.909863 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "kube-api-access-sn6k8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:39.910028 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:39.909905 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token" pod "37723510-9540-4794-b253-9e85fc0f6846" (UID: "37723510-9540-4794-b253-9e85fc0f6846"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 05:59:40.006621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006590 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn6k8\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-kube-api-access-sn6k8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006619 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006621 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006630 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37723510-9540-4794-b253-9e85fc0f6846-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006638 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006647 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006656 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37723510-9540-4794-b253-9e85fc0f6846-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006663 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37723510-9540-4794-b253-9e85fc0f6846-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.006830 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.006671 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37723510-9540-4794-b253-9e85fc0f6846-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 05:59:40.386661 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.386638 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" Oct 30 05:59:40.387098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.386688 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2" event={"ID":"140e5755-d9a3-428f-8bd2-616cabc92c2a","Type":"ContainerDied","Data":"6ca4dbf5d6b2a2b87b880c1d09bea4ff0e0f33a691666130e1c1be3d3a48ba7d"} Oct 30 05:59:40.387098 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.386747 5565 scope.go:122] "RemoveContainer" containerID="b2aee50c4fe9897573e4b077c60c0cb444f03dd8d02653c56598785cdac01a92" Oct 30 05:59:40.388520 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.388496 5565 generic.go:356] "Generic (PLEG): container finished" podID="37723510-9540-4794-b253-9e85fc0f6846" containerID="2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e" exitCode=0 Oct 30 05:59:40.388681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.388565 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" event={"ID":"37723510-9540-4794-b253-9e85fc0f6846","Type":"ContainerDied","Data":"2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e"} Oct 30 05:59:40.388681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.388584 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" Oct 30 05:59:40.388681 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.388619 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd" event={"ID":"37723510-9540-4794-b253-9e85fc0f6846","Type":"ContainerDied","Data":"ba1e192ec5a1b2b169dd060e564101217a6da650d6673faa5a978e3437a7e96b"} Oct 30 05:59:40.402391 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.402326 5565 scope.go:122] "RemoveContainer" containerID="2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e" Oct 30 05:59:40.413219 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.413199 5565 scope.go:122] "RemoveContainer" containerID="2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e" Oct 30 05:59:40.413502 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:59:40.413475 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e\": container with ID starting with 2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e not found: ID does not exist" containerID="2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e" Oct 30 05:59:40.413654 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.413506 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e"} err="failed to get container status \"2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e\": rpc error: code = NotFound desc = could not find container \"2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e\": container with ID starting with 2842f2e642fc20d1628454baeac00755e6722edd46d7c14056851064cf9c1a9e not found: ID does not exist" Oct 30 05:59:40.465147 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.465115 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd"] Oct 30 05:59:40.478945 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.478919 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-bgrrd"] Oct 30 05:59:40.509867 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.509843 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2"] Oct 30 05:59:40.515769 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:40.515746 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-6bjx2"] Oct 30 05:59:41.640548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:41.632861 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" path="/var/lib/kubelet/pods/140e5755-d9a3-428f-8bd2-616cabc92c2a/volumes" Oct 30 05:59:41.640548 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:41.633747 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37723510-9540-4794-b253-9e85fc0f6846" path="/var/lib/kubelet/pods/37723510-9540-4794-b253-9e85fc0f6846/volumes" Oct 30 05:59:42.483406 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:42.483372 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc"] Oct 30 05:59:42.497179 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:42.497153 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-76b905-9m6mc"] Oct 30 05:59:43.572489 ip-10-0-129-175 kubenswrapper[5565]: I1030 05:59:43.572451 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4baa3f0-f9da-4c06-a756-a0bae00da117" path="/var/lib/kubelet/pods/e4baa3f0-f9da-4c06-a756-a0bae00da117/volumes" Oct 30 05:59:47.534312 ip-10-0-129-175 kubenswrapper[5565]: E1030 05:59:47.534269 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:00:00.175418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.175383 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m"] Oct 30 06:00:00.176166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176149 5565 cpu_manager.go:414] "containerMap: removing container" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" Oct 30 06:00:00.176208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176167 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" Oct 30 06:00:00.176208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176183 5565 cpu_manager.go:414] "containerMap: removing container" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" Oct 30 06:00:00.176208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176188 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" Oct 30 06:00:00.176208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176205 5565 cpu_manager.go:414] "containerMap: removing container" podUID="20edd61a-214b-42bd-a4cb-df979476fa73" containerName="revoke-keys" Oct 30 06:00:00.176367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176211 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20edd61a-214b-42bd-a4cb-df979476fa73" containerName="revoke-keys" Oct 30 06:00:00.176367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176228 5565 cpu_manager.go:414] "containerMap: removing container" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" Oct 30 06:00:00.176367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176233 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" Oct 30 06:00:00.176367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176350 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="37723510-9540-4794-b253-9e85fc0f6846" containerName="istio-proxy" Oct 30 06:00:00.176367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176362 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="20edd61a-214b-42bd-a4cb-df979476fa73" containerName="revoke-keys" Oct 30 06:00:00.176516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176369 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f54885-68d7-439b-b786-d314d7af6eb0" containerName="istio-proxy" Oct 30 06:00:00.176516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.176384 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="140e5755-d9a3-428f-8bd2-616cabc92c2a" containerName="istio-proxy" Oct 30 06:00:00.180363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.180331 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg"] Oct 30 06:00:00.180472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.180439 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.184906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.184883 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2"] Oct 30 06:00:00.185014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.184992 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.189477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.189457 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.224660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.224637 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm"] Oct 30 06:00:00.229778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.229755 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c"] Oct 30 06:00:00.230048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.230030 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.249550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.241600 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg"] Oct 30 06:00:00.249550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.241701 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.303063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.303032 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:00:00.313600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.313568 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m"] Oct 30 06:00:00.316165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.316138 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2"] Oct 30 06:00:00.326625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.326596 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp"] Oct 30 06:00:00.329088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329058 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq7vf\" (UniqueName: \"kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.329219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329157 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt5t2\" (UniqueName: \"kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.329219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329193 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7m7m\" (UniqueName: \"kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m\") pod \"maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.329352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329238 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.329411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329348 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.329476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329444 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.329580 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329561 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.329664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329607 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98sxr\" (UniqueName: \"kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.329664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329644 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8g8w\" (UniqueName: \"kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.329797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.329740 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.340574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.338405 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn"] Oct 30 06:00:00.340574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.339809 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.355670 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.355648 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm"] Oct 30 06:00:00.355807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.355749 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.364116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.364075 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m"] Oct 30 06:00:00.370812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.370788 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp"] Oct 30 06:00:00.370935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.370895 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.374400 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.374379 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv"] Oct 30 06:00:00.380926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.380907 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn"] Oct 30 06:00:00.381076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.381001 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.383066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.383047 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q"] Oct 30 06:00:00.390036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.390014 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m"] Oct 30 06:00:00.390165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.390120 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.391539 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.391494 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-086c60-openshift-default-dockercfg-rnptw\"" Oct 30 06:00:00.393736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.393626 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx"] Oct 30 06:00:00.418001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.417972 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv"] Oct 30 06:00:00.418153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.418030 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c"] Oct 30 06:00:00.418153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.418137 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.427314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.427247 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q"] Oct 30 06:00:00.431604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431565 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.431740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431648 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq7vf\" (UniqueName: \"kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.431740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431718 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.431860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431773 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt5t2\" (UniqueName: \"kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.431860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431810 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7m7m\" (UniqueName: \"kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m\") pod \"maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.431966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431859 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.431966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431898 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.432146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.431986 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.432146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432050 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.432146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432110 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.432310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432170 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.432310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432220 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.432310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432253 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889000-rzrtp\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.432469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432310 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm9sj\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.432469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432382 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqld7\" (UniqueName: \"kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.432469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432446 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.432657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432489 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.432657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432558 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98sxr\" (UniqueName: \"kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.432657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432592 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.432657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432636 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.432871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432671 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8g8w\" (UniqueName: \"kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.432871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432785 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxkr9\" (UniqueName: \"kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9\") pod \"maas-api-key-cleanup-29889000-rzrtp\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.432871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.432837 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.433732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.433704 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.435165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.435142 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.435846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.435818 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.436598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.436571 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.437300 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.437273 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx"] Oct 30 06:00:00.438500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.437950 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.463132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.463079 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7m7m\" (UniqueName: \"kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m\") pod \"maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.463245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.463176 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq7vf\" (UniqueName: \"kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf\") pod \"maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.464542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.464282 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98sxr\" (UniqueName: \"kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.464542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.464487 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt5t2\" (UniqueName: \"kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.464967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.464944 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8g8w\" (UniqueName: \"kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w\") pod \"maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.496284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.496264 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:00.512694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.512669 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:00.516500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.516476 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:00.535094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535057 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dcn6\" (UniqueName: \"kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.535269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535108 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.535269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535194 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2bc4\" (UniqueName: \"kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4\") pod \"maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.535269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535248 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.535466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535310 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.535466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535342 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.535466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535394 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.535466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535442 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889000-rzrtp\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.535743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535489 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm9sj\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.535743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535564 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqld7\" (UniqueName: \"kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.535743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535610 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.535743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535655 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.535743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535697 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.535743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535734 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.536068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535845 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxkr9\" (UniqueName: \"kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9\") pod \"maas-api-key-cleanup-29889000-rzrtp\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.536068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.535918 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.536068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.536011 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.536230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.536081 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.536230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.536120 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.537803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.536997 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.537803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.537262 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889000-rzrtp\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.537803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.537764 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.538047 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.537983 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.538113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.538066 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.538172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.538118 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.538576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.538516 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.540549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.540496 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.542520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.542502 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.569089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.568883 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxkr9\" (UniqueName: \"kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9\") pod \"maas-api-key-cleanup-29889000-rzrtp\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.570208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.570145 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm9sj\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.571387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.570928 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.571387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.571347 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqld7\" (UniqueName: \"kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.620199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.619825 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:00.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.638092 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dcn6\" (UniqueName: \"kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.638173 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2bc4\" (UniqueName: \"kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4\") pod \"maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.638226 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.638301 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.638483 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.638999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.638889 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.641992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.641947 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.643054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.643024 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.643159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.643072 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.657113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.657093 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:00.666681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.666158 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:00.674652 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.674383 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2bc4\" (UniqueName: \"kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4\") pod \"maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.674841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.674777 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dcn6\" (UniqueName: \"kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.681251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.681201 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:00.697152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.696592 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:00.727760 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.727731 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:00.743990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.741544 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:00.754295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.753991 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:00.762876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:00.762795 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:01.026281 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:01.026251 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b7679c5_5604_4cd1_ada4_ff8abf2c136f.slice/crio-c5b340bda9051a0a9e167724a2a764d02ee00305ca4e445ea9a1889da7e5af28 WatchSource:0}: Error finding container c5b340bda9051a0a9e167724a2a764d02ee00305ca4e445ea9a1889da7e5af28: Status 404 returned error can't find the container with id c5b340bda9051a0a9e167724a2a764d02ee00305ca4e445ea9a1889da7e5af28 Oct 30 06:00:01.029508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.029477 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m"] Oct 30 06:00:01.156699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.156666 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm"] Oct 30 06:00:01.159600 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:01.159557 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7942a8bf_2038_412f_b16d_aa9e1859bf58.slice/crio-f81568dd7ec51f262d5b93287548940b6e1b794ee7f6d9cde00a25639f67f0d1 WatchSource:0}: Error finding container f81568dd7ec51f262d5b93287548940b6e1b794ee7f6d9cde00a25639f67f0d1: Status 404 returned error can't find the container with id f81568dd7ec51f262d5b93287548940b6e1b794ee7f6d9cde00a25639f67f0d1 Oct 30 06:00:01.212134 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.212109 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv"] Oct 30 06:00:01.225211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.225164 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx"] Oct 30 06:00:01.227266 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:01.227239 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04d1cb96_7416_41e9_8a1b_61544771a98b.slice/crio-c4f033abe6a943a51872d9aba3f3d324b76be8d44d4148433c6a9692c3c23c2c WatchSource:0}: Error finding container c4f033abe6a943a51872d9aba3f3d324b76be8d44d4148433c6a9692c3c23c2c: Status 404 returned error can't find the container with id c4f033abe6a943a51872d9aba3f3d324b76be8d44d4148433c6a9692c3c23c2c Oct 30 06:00:01.230605 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:01.230579 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddd289bd_175a_432d_8f42_6eba3a427876.slice/crio-93fb25cda9e55ce44edd205eec35a870f626bc01f13c739463655d494f69c65d WatchSource:0}: Error finding container 93fb25cda9e55ce44edd205eec35a870f626bc01f13c739463655d494f69c65d: Status 404 returned error can't find the container with id 93fb25cda9e55ce44edd205eec35a870f626bc01f13c739463655d494f69c65d Oct 30 06:00:01.231040 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.231013 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c"] Oct 30 06:00:01.247655 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:01.247630 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfc44cce_61f3_4c99_b90a_6edee0789b80.slice/crio-1da28346e9efa93ebc8983a8b98f9c43d8efa41e3856b1936147b7caccad5f2a WatchSource:0}: Error finding container 1da28346e9efa93ebc8983a8b98f9c43d8efa41e3856b1936147b7caccad5f2a: Status 404 returned error can't find the container with id 1da28346e9efa93ebc8983a8b98f9c43d8efa41e3856b1936147b7caccad5f2a Oct 30 06:00:01.253084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.253058 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg"] Oct 30 06:00:01.269864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.269754 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp"] Oct 30 06:00:01.276310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.276282 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn"] Oct 30 06:00:01.303342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.303318 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2"] Oct 30 06:00:01.572913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.572860 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:01.573174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.573145 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:01.573348 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.573328 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:01.666139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.666118 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q"] Oct 30 06:00:01.672969 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.672949 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m"] Oct 30 06:00:01.717051 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.716807 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" event={"ID":"09e41011-96e4-4203-9cf7-83c8a991aff1","Type":"ContainerStarted","Data":"7164932ea8326820c6267224f9f21d6c2fec1b4b4a9e8b93aa06ebce8a7f3e62"} Oct 30 06:00:01.718280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.718253 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" event={"ID":"043a6d14-fd3b-409a-8480-e5e6dd077eb1","Type":"ContainerStarted","Data":"b27f1ade3c33a140e98a02cbb8f8c13e65e43143f7e41e4ac4586ef7fe9bf3fd"} Oct 30 06:00:01.720000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.719977 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerStarted","Data":"93fb25cda9e55ce44edd205eec35a870f626bc01f13c739463655d494f69c65d"} Oct 30 06:00:01.722984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.722957 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerStarted","Data":"f74b2ac52262557708d123d9e5f4d2d74b7c9647033fbcf057b0c77696d47079"} Oct 30 06:00:01.723164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.723140 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerStarted","Data":"f81568dd7ec51f262d5b93287548940b6e1b794ee7f6d9cde00a25639f67f0d1"} Oct 30 06:00:01.753779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.750913 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerStarted","Data":"b03ad62b71f323e48d345f804034d32ff5a1f83436264a3c32639729b70b9b56"} Oct 30 06:00:01.763298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.763210 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerStarted","Data":"1da28346e9efa93ebc8983a8b98f9c43d8efa41e3856b1936147b7caccad5f2a"} Oct 30 06:00:01.770142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.769020 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerStarted","Data":"3025107b3c46c173ea1e25ce14701f85b3d3a17259684ccd25e658b5727a6251"} Oct 30 06:00:01.770142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.769092 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerStarted","Data":"c5b340bda9051a0a9e167724a2a764d02ee00305ca4e445ea9a1889da7e5af28"} Oct 30 06:00:01.780117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.779987 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerStarted","Data":"c4f033abe6a943a51872d9aba3f3d324b76be8d44d4148433c6a9692c3c23c2c"} Oct 30 06:00:01.813775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.811142 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerStarted","Data":"1cc3fd5f52e1ae8db432611f422408728e7cc850ed2901e09757f46152d7526c"} Oct 30 06:00:01.819022 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.818991 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerStarted","Data":"949cf3918c3e560dbae4b7438c06ef3b23701858905d656e1cc2bd3cf9ff4279"} Oct 30 06:00:01.820501 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.820445 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" podStartSLOduration=1.820427314 podStartE2EDuration="1.820427314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:01.807646323 +0000 UTC m=+1708.383333378" watchObservedRunningTime="2026-10-30 06:00:01.820427314 +0000 UTC m=+1708.396114376" Oct 30 06:00:01.821017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.820988 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerStarted","Data":"86ee8dd9f0522747d4d3d97b48ca004085cdf68a091b4053c1c2df30ca7bb571"} Oct 30 06:00:01.863952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:01.863900 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" podStartSLOduration=1.863885239 podStartE2EDuration="1.863885239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:01.860808744 +0000 UTC m=+1708.436495801" watchObservedRunningTime="2026-10-30 06:00:01.863885239 +0000 UTC m=+1708.439572292" Oct 30 06:00:02.832190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.832113 5565 generic.go:356] "Generic (PLEG): container finished" podID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerID="1a4c96ad4a4079b910bbc79844e78d12ade2ed25c878f9b9dacc717959498411" exitCode=60 Oct 30 06:00:02.832642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.832216 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerDied","Data":"1a4c96ad4a4079b910bbc79844e78d12ade2ed25c878f9b9dacc717959498411"} Oct 30 06:00:02.832642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.832581 5565 scope.go:122] "RemoveContainer" containerID="1a4c96ad4a4079b910bbc79844e78d12ade2ed25c878f9b9dacc717959498411" Oct 30 06:00:02.834077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.834054 5565 generic.go:356] "Generic (PLEG): container finished" podID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerID="cc1a6fcba299f8ad26cb3993ab3eeaba600e4536da61f6c82f8aa5af9205f8d0" exitCode=60 Oct 30 06:00:02.834190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.834127 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerDied","Data":"cc1a6fcba299f8ad26cb3993ab3eeaba600e4536da61f6c82f8aa5af9205f8d0"} Oct 30 06:00:02.834439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.834379 5565 scope.go:122] "RemoveContainer" containerID="cc1a6fcba299f8ad26cb3993ab3eeaba600e4536da61f6c82f8aa5af9205f8d0" Oct 30 06:00:02.836074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.836021 5565 generic.go:356] "Generic (PLEG): container finished" podID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerID="3025107b3c46c173ea1e25ce14701f85b3d3a17259684ccd25e658b5727a6251" exitCode=60 Oct 30 06:00:02.836074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.836051 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerDied","Data":"3025107b3c46c173ea1e25ce14701f85b3d3a17259684ccd25e658b5727a6251"} Oct 30 06:00:02.836419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.836403 5565 scope.go:122] "RemoveContainer" containerID="3025107b3c46c173ea1e25ce14701f85b3d3a17259684ccd25e658b5727a6251" Oct 30 06:00:02.837974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.837951 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerStarted","Data":"e1ded4713c14a23f18ed91190ecd6a40baef48cd43db445bc96b3acf0c3bbdae"} Oct 30 06:00:02.839898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.839876 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerStarted","Data":"72d6c58281c20153f746d54ba8d92105452fe3a625534c13874dda5c8291a6b9"} Oct 30 06:00:02.841508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.841483 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerStarted","Data":"194f410be19b412c3992ae9839a4e1266a4a041461d060e1b1eda6597859b3f0"} Oct 30 06:00:02.843097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.843074 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerStarted","Data":"9cd0f7bd5a3ab61713222d4a6f575869d3d6f4d2089fdb4c30e8d25ad3694c21"} Oct 30 06:00:02.844833 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.844810 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" event={"ID":"09e41011-96e4-4203-9cf7-83c8a991aff1","Type":"ContainerStarted","Data":"b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d"} Oct 30 06:00:02.852493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.852470 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" event={"ID":"043a6d14-fd3b-409a-8480-e5e6dd077eb1","Type":"ContainerStarted","Data":"0d2d17f51ca93b07127a62be5d0980d0a9dec642ec4761531d31200bde169231"} Oct 30 06:00:02.878722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.877822 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerStarted","Data":"19c38eaba0c66bb2e01b3ca982a6a8f31da5a36de54bc01e0f67a58838714ca9"} Oct 30 06:00:02.896826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.896781 5565 generic.go:356] "Generic (PLEG): container finished" podID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerID="f74b2ac52262557708d123d9e5f4d2d74b7c9647033fbcf057b0c77696d47079" exitCode=60 Oct 30 06:00:02.897011 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.896857 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerDied","Data":"f74b2ac52262557708d123d9e5f4d2d74b7c9647033fbcf057b0c77696d47079"} Oct 30 06:00:02.898112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.898088 5565 scope.go:122] "RemoveContainer" containerID="f74b2ac52262557708d123d9e5f4d2d74b7c9647033fbcf057b0c77696d47079" Oct 30 06:00:02.926719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:02.926675 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" podStartSLOduration=2.926662086 podStartE2EDuration="2.926662086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:02.923752919 +0000 UTC m=+1709.499439974" watchObservedRunningTime="2026-10-30 06:00:02.926662086 +0000 UTC m=+1709.502349140" Oct 30 06:00:03.022609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.022556 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" podStartSLOduration=3.022520816 podStartE2EDuration="3.022520816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.020319638 +0000 UTC m=+1709.596006692" watchObservedRunningTime="2026-10-30 06:00:03.022520816 +0000 UTC m=+1709.598207871" Oct 30 06:00:03.129478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.129429 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podStartSLOduration=3.12941481 podStartE2EDuration="3.12941481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.081944468 +0000 UTC m=+1709.657631522" watchObservedRunningTime="2026-10-30 06:00:03.12941481 +0000 UTC m=+1709.705101864" Oct 30 06:00:03.131232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.131198 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" podStartSLOduration=3.13118585 podStartE2EDuration="3.13118585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.12690318 +0000 UTC m=+1709.702590235" watchObservedRunningTime="2026-10-30 06:00:03.13118585 +0000 UTC m=+1709.706872903" Oct 30 06:00:03.181297 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.181239 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" podStartSLOduration=3.181219396 podStartE2EDuration="3.181219396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.178981879 +0000 UTC m=+1709.754668934" watchObservedRunningTime="2026-10-30 06:00:03.181219396 +0000 UTC m=+1709.756906451" Oct 30 06:00:03.282164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.282102 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" podStartSLOduration=3.282077349 podStartE2EDuration="3.282077349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.268542936 +0000 UTC m=+1709.844229986" watchObservedRunningTime="2026-10-30 06:00:03.282077349 +0000 UTC m=+1709.857764429" Oct 30 06:00:03.317851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.317805 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" podStartSLOduration=3.317791769 podStartE2EDuration="3.317791769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:03.313803057 +0000 UTC m=+1709.889490112" watchObservedRunningTime="2026-10-30 06:00:03.317791769 +0000 UTC m=+1709.893478823" Oct 30 06:00:03.697893 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.697774 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:03.699144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.699118 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:00:03.699254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.699162 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:00:03.920953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.920918 5565 generic.go:356] "Generic (PLEG): container finished" podID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerID="0d2d17f51ca93b07127a62be5d0980d0a9dec642ec4761531d31200bde169231" exitCode=60 Oct 30 06:00:03.921308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.920988 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" event={"ID":"043a6d14-fd3b-409a-8480-e5e6dd077eb1","Type":"ContainerDied","Data":"0d2d17f51ca93b07127a62be5d0980d0a9dec642ec4761531d31200bde169231"} Oct 30 06:00:03.921345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.921315 5565 scope.go:122] "RemoveContainer" containerID="0d2d17f51ca93b07127a62be5d0980d0a9dec642ec4761531d31200bde169231" Oct 30 06:00:03.923588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.923566 5565 generic.go:356] "Generic (PLEG): container finished" podID="ddd289bd-175a-432d-8f42-6eba3a427876" containerID="19c38eaba0c66bb2e01b3ca982a6a8f31da5a36de54bc01e0f67a58838714ca9" exitCode=60 Oct 30 06:00:03.923680 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.923641 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerDied","Data":"19c38eaba0c66bb2e01b3ca982a6a8f31da5a36de54bc01e0f67a58838714ca9"} Oct 30 06:00:03.923935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.923918 5565 scope.go:122] "RemoveContainer" containerID="19c38eaba0c66bb2e01b3ca982a6a8f31da5a36de54bc01e0f67a58838714ca9" Oct 30 06:00:03.927855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.927196 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerStarted","Data":"ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab"} Oct 30 06:00:03.931394 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.930786 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerStarted","Data":"9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643"} Oct 30 06:00:03.933538 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.933499 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerStarted","Data":"63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1"} Oct 30 06:00:03.936897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.936298 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerStarted","Data":"155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026"} Oct 30 06:00:03.942464 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.942435 5565 generic.go:356] "Generic (PLEG): container finished" podID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerID="e1ded4713c14a23f18ed91190ecd6a40baef48cd43db445bc96b3acf0c3bbdae" exitCode=60 Oct 30 06:00:03.942616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.942519 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerDied","Data":"e1ded4713c14a23f18ed91190ecd6a40baef48cd43db445bc96b3acf0c3bbdae"} Oct 30 06:00:03.942916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.942896 5565 scope.go:122] "RemoveContainer" containerID="e1ded4713c14a23f18ed91190ecd6a40baef48cd43db445bc96b3acf0c3bbdae" Oct 30 06:00:03.952350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.952313 5565 generic.go:356] "Generic (PLEG): container finished" podID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerID="72d6c58281c20153f746d54ba8d92105452fe3a625534c13874dda5c8291a6b9" exitCode=60 Oct 30 06:00:03.952443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.952396 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerDied","Data":"72d6c58281c20153f746d54ba8d92105452fe3a625534c13874dda5c8291a6b9"} Oct 30 06:00:03.986566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:03.952753 5565 scope.go:122] "RemoveContainer" containerID="72d6c58281c20153f746d54ba8d92105452fe3a625534c13874dda5c8291a6b9" Oct 30 06:00:04.551548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.543301 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z"] Oct 30 06:00:04.613691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.613479 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerID="194f410be19b412c3992ae9839a4e1266a4a041461d060e1b1eda6597859b3f0" exitCode=60 Oct 30 06:00:04.613691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.613487 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerDied","Data":"194f410be19b412c3992ae9839a4e1266a4a041461d060e1b1eda6597859b3f0"} Oct 30 06:00:04.614592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.614298 5565 scope.go:122] "RemoveContainer" containerID="194f410be19b412c3992ae9839a4e1266a4a041461d060e1b1eda6597859b3f0" Oct 30 06:00:04.626390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.624886 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerDied","Data":"9cd0f7bd5a3ab61713222d4a6f575869d3d6f4d2089fdb4c30e8d25ad3694c21"} Oct 30 06:00:04.626390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.624885 5565 generic.go:356] "Generic (PLEG): container finished" podID="7436eeec-3732-4338-9c91-373887957bde" containerID="9cd0f7bd5a3ab61713222d4a6f575869d3d6f4d2089fdb4c30e8d25ad3694c21" exitCode=60 Oct 30 06:00:04.626390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.625427 5565 scope.go:122] "RemoveContainer" containerID="9cd0f7bd5a3ab61713222d4a6f575869d3d6f4d2089fdb4c30e8d25ad3694c21" Oct 30 06:00:04.626994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.626874 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-386c4e11-qmq9z"] Oct 30 06:00:04.650695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.650640 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" podStartSLOduration=4.650623466 podStartE2EDuration="4.650623466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.564407641 +0000 UTC m=+1711.140094697" watchObservedRunningTime="2026-10-30 06:00:04.650623466 +0000 UTC m=+1711.226310521" Oct 30 06:00:04.663046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.662996 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" podStartSLOduration=4.66298245 podStartE2EDuration="4.66298245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:04.617220295 +0000 UTC m=+1711.192907369" watchObservedRunningTime="2026-10-30 06:00:04.66298245 +0000 UTC m=+1711.238669505" Oct 30 06:00:04.697740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.697707 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:00:04.697886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:04.697769 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:00:05.589370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.589340 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3605ca7-fb50-435a-9000-8f8fe0246273" path="/var/lib/kubelet/pods/f3605ca7-fb50-435a-9000-8f8fe0246273/volumes" Oct 30 06:00:05.635307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.635270 5565 generic.go:356] "Generic (PLEG): container finished" podID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerID="155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026" exitCode=60 Oct 30 06:00:05.635452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.635353 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerDied","Data":"155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026"} Oct 30 06:00:05.635452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.635409 5565 scope.go:122] "RemoveContainer" containerID="3025107b3c46c173ea1e25ce14701f85b3d3a17259684ccd25e658b5727a6251" Oct 30 06:00:05.635980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.635715 5565 scope.go:122] "RemoveContainer" containerID="155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026" Oct 30 06:00:05.636139 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:05.636115 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m_odh-ai-gateway-infra(6b7679c5-5604-4cd1-ada4-ff8abf2c136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" Oct 30 06:00:05.638685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.638655 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerStarted","Data":"e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86"} Oct 30 06:00:05.641609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.641582 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerStarted","Data":"2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05"} Oct 30 06:00:05.643809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.643772 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerStarted","Data":"91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30"} Oct 30 06:00:05.646324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.646167 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerStarted","Data":"de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf"} Oct 30 06:00:05.648703 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.648683 5565 generic.go:356] "Generic (PLEG): container finished" podID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerID="ff3cd4708b4ee9b00c904bff0237792866b3d95f93ba157be2581e578f3f5978" exitCode=60 Oct 30 06:00:05.648790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.648749 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" event={"ID":"043a6d14-fd3b-409a-8480-e5e6dd077eb1","Type":"ContainerDied","Data":"ff3cd4708b4ee9b00c904bff0237792866b3d95f93ba157be2581e578f3f5978"} Oct 30 06:00:05.649020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.649003 5565 scope.go:122] "RemoveContainer" containerID="ff3cd4708b4ee9b00c904bff0237792866b3d95f93ba157be2581e578f3f5978" Oct 30 06:00:05.649328 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:05.649303 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m_odh-ai-gateway-infra(043a6d14-fd3b-409a-8480-e5e6dd077eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" Oct 30 06:00:05.650730 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.650701 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerStarted","Data":"fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e"} Oct 30 06:00:05.663599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.652510 5565 generic.go:356] "Generic (PLEG): container finished" podID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerID="ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab" exitCode=60 Oct 30 06:00:05.663599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.652646 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerDied","Data":"ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab"} Oct 30 06:00:05.663599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.659028 5565 scope.go:122] "RemoveContainer" containerID="ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab" Oct 30 06:00:05.663599 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:05.659324 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm_odh-ai-gateway-infra(7942a8bf-2038-412f-b16d-aa9e1859bf58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" Oct 30 06:00:05.663599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.660904 5565 scope.go:122] "RemoveContainer" containerID="0d2d17f51ca93b07127a62be5d0980d0a9dec642ec4761531d31200bde169231" Oct 30 06:00:05.676659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.675970 5565 generic.go:356] "Generic (PLEG): container finished" podID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerID="9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643" exitCode=60 Oct 30 06:00:05.676659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.676086 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerDied","Data":"9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643"} Oct 30 06:00:05.676659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.676516 5565 scope.go:122] "RemoveContainer" containerID="9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643" Oct 30 06:00:05.687605 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:05.676895 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv_odh-ai-gateway-infra(542230a4-fb15-4a4d-bc64-d77c8b4d3806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" Oct 30 06:00:05.687605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.683501 5565 generic.go:356] "Generic (PLEG): container finished" podID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerID="63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1" exitCode=60 Oct 30 06:00:05.687605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.683669 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerDied","Data":"63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1"} Oct 30 06:00:05.687605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.684016 5565 scope.go:122] "RemoveContainer" containerID="63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1" Oct 30 06:00:05.687605 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:05.684317 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg_odh-ai-gateway-infra(cfc44cce-61f3-4c99-b90a-6edee0789b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" Oct 30 06:00:05.707605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.697060 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:00:05.707605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.697105 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:00:05.707605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:05.699205 5565 scope.go:122] "RemoveContainer" containerID="f74b2ac52262557708d123d9e5f4d2d74b7c9647033fbcf057b0c77696d47079" Oct 30 06:00:06.099404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.096747 5565 scope.go:122] "RemoveContainer" containerID="1a4c96ad4a4079b910bbc79844e78d12ade2ed25c878f9b9dacc717959498411" Oct 30 06:00:06.118477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.118451 5565 scope.go:122] "RemoveContainer" containerID="cc1a6fcba299f8ad26cb3993ab3eeaba600e4536da61f6c82f8aa5af9205f8d0" Oct 30 06:00:06.695357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.695326 5565 scope.go:122] "RemoveContainer" containerID="155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026" Oct 30 06:00:06.695864 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.695784 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m_odh-ai-gateway-infra(6b7679c5-5604-4cd1-ada4-ff8abf2c136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" Oct 30 06:00:06.697137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.697106 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:00:06.697271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.697159 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:00:06.697578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.697554 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerDied","Data":"e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86"} Oct 30 06:00:06.697637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.697614 5565 scope.go:122] "RemoveContainer" containerID="e1ded4713c14a23f18ed91190ecd6a40baef48cd43db445bc96b3acf0c3bbdae" Oct 30 06:00:06.697693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.697514 5565 generic.go:356] "Generic (PLEG): container finished" podID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerID="e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86" exitCode=60 Oct 30 06:00:06.697900 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.697878 5565 scope.go:122] "RemoveContainer" containerID="e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86" Oct 30 06:00:06.698264 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.698235 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx_odh-ai-gateway-infra(04d1cb96-7416-41e9-8a1b-61544771a98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" Oct 30 06:00:06.704848 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.704827 5565 generic.go:356] "Generic (PLEG): container finished" podID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerID="2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05" exitCode=60 Oct 30 06:00:06.704932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.704908 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerDied","Data":"2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05"} Oct 30 06:00:06.705154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.705140 5565 scope.go:122] "RemoveContainer" containerID="2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05" Oct 30 06:00:06.705439 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.705417 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q_odh-ai-gateway-infra(940e5c05-e3c9-4074-bab3-21d1771bbdcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" Oct 30 06:00:06.706981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.706960 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerID="91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30" exitCode=60 Oct 30 06:00:06.707263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.707246 5565 scope.go:122] "RemoveContainer" containerID="91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30" Oct 30 06:00:06.707488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.707036 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerDied","Data":"91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30"} Oct 30 06:00:06.707588 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.707506 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2_odh-ai-gateway-infra(f5be034e-5537-4b8b-8f9b-6163c306fa87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" Oct 30 06:00:06.709178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.709159 5565 generic.go:356] "Generic (PLEG): container finished" podID="7436eeec-3732-4338-9c91-373887957bde" containerID="de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf" exitCode=60 Oct 30 06:00:06.709276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.709221 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerDied","Data":"de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf"} Oct 30 06:00:06.709429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.709411 5565 scope.go:122] "RemoveContainer" containerID="de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf" Oct 30 06:00:06.709754 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.709725 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889000-rzrtp_odh-ai-gateway-infra(7436eeec-3732-4338-9c91-373887957bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" podUID="7436eeec-3732-4338-9c91-373887957bde" Oct 30 06:00:06.711318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.711304 5565 scope.go:122] "RemoveContainer" containerID="ff3cd4708b4ee9b00c904bff0237792866b3d95f93ba157be2581e578f3f5978" Oct 30 06:00:06.711592 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.711573 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m_odh-ai-gateway-infra(043a6d14-fd3b-409a-8480-e5e6dd077eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" Oct 30 06:00:06.712934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.712915 5565 generic.go:356] "Generic (PLEG): container finished" podID="ddd289bd-175a-432d-8f42-6eba3a427876" containerID="fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e" exitCode=60 Oct 30 06:00:06.713037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.712980 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerDied","Data":"fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e"} Oct 30 06:00:06.713181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.713166 5565 scope.go:122] "RemoveContainer" containerID="fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e" Oct 30 06:00:06.713432 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.713408 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c_odh-ai-gateway-infra(ddd289bd-175a-432d-8f42-6eba3a427876)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" Oct 30 06:00:06.714825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.714804 5565 scope.go:122] "RemoveContainer" containerID="ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab" Oct 30 06:00:06.715074 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.715055 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm_odh-ai-gateway-infra(7942a8bf-2038-412f-b16d-aa9e1859bf58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" Oct 30 06:00:06.716614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.716597 5565 scope.go:122] "RemoveContainer" containerID="9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643" Oct 30 06:00:06.716861 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.716843 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv_odh-ai-gateway-infra(542230a4-fb15-4a4d-bc64-d77c8b4d3806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" Oct 30 06:00:06.718295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.718246 5565 scope.go:122] "RemoveContainer" containerID="63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1" Oct 30 06:00:06.718346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.718322 5565 scope.go:122] "RemoveContainer" containerID="72d6c58281c20153f746d54ba8d92105452fe3a625534c13874dda5c8291a6b9" Oct 30 06:00:06.718491 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:06.718469 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg_odh-ai-gateway-infra(cfc44cce-61f3-4c99-b90a-6edee0789b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" Oct 30 06:00:06.732639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.732627 5565 scope.go:122] "RemoveContainer" containerID="194f410be19b412c3992ae9839a4e1266a4a041461d060e1b1eda6597859b3f0" Oct 30 06:00:06.748146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.748124 5565 scope.go:122] "RemoveContainer" containerID="9cd0f7bd5a3ab61713222d4a6f575869d3d6f4d2089fdb4c30e8d25ad3694c21" Oct 30 06:00:06.782552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:06.778705 5565 scope.go:122] "RemoveContainer" containerID="19c38eaba0c66bb2e01b3ca982a6a8f31da5a36de54bc01e0f67a58838714ca9" Oct 30 06:00:07.716598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.715504 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body= Oct 30 06:00:07.716598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.715581 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" Oct 30 06:00:07.744158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.744133 5565 scope.go:122] "RemoveContainer" containerID="fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e" Oct 30 06:00:07.746195 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:07.746162 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c_odh-ai-gateway-infra(ddd289bd-175a-432d-8f42-6eba3a427876)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" Oct 30 06:00:07.747726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.747706 5565 scope.go:122] "RemoveContainer" containerID="e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86" Oct 30 06:00:07.748213 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:07.748192 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx_odh-ai-gateway-infra(04d1cb96-7416-41e9-8a1b-61544771a98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" Oct 30 06:00:07.750455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.750430 5565 scope.go:122] "RemoveContainer" containerID="2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05" Oct 30 06:00:07.750843 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:07.750813 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q_odh-ai-gateway-infra(940e5c05-e3c9-4074-bab3-21d1771bbdcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" Oct 30 06:00:07.784783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.769586 5565 scope.go:122] "RemoveContainer" containerID="91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30" Oct 30 06:00:07.785345 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:07.785289 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2_odh-ai-gateway-infra(f5be034e-5537-4b8b-8f9b-6163c306fa87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" Oct 30 06:00:07.788710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:07.788678 5565 scope.go:122] "RemoveContainer" containerID="de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf" Oct 30 06:00:07.789125 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:07.789099 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889000-rzrtp_odh-ai-gateway-infra(7436eeec-3732-4338-9c91-373887957bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" podUID="7436eeec-3732-4338-9c91-373887957bde" Oct 30 06:00:08.702840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:08.702810 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:08.703284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:08.703255 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:08.704092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:08.704072 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:00:15.578461 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:15.578417 5565 scope.go:122] "RemoveContainer" containerID="ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab" Oct 30 06:00:15.579012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:15.578666 5565 scope.go:122] "RemoveContainer" containerID="155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026" Oct 30 06:00:16.560913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.560881 5565 scope.go:122] "RemoveContainer" containerID="fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e" Oct 30 06:00:16.561077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.560985 5565 scope.go:122] "RemoveContainer" containerID="ff3cd4708b4ee9b00c904bff0237792866b3d95f93ba157be2581e578f3f5978" Oct 30 06:00:16.891220 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.891188 5565 generic.go:356] "Generic (PLEG): container finished" podID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerID="892b3dace8106ac65d529d19d8e3a7ec792f5a2e11380586cb3f7351ff350500" exitCode=60 Oct 30 06:00:16.891746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.891258 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerDied","Data":"892b3dace8106ac65d529d19d8e3a7ec792f5a2e11380586cb3f7351ff350500"} Oct 30 06:00:16.891746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.891326 5565 scope.go:122] "RemoveContainer" containerID="155e69937c4f3fd87d9aa14d7cf86006fbeb113aa9e9f62718d401694952b026" Oct 30 06:00:16.891916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.891892 5565 scope.go:122] "RemoveContainer" containerID="892b3dace8106ac65d529d19d8e3a7ec792f5a2e11380586cb3f7351ff350500" Oct 30 06:00:16.892275 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:16.892251 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m_odh-ai-gateway-infra(6b7679c5-5604-4cd1-ada4-ff8abf2c136f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" Oct 30 06:00:16.894880 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.894853 5565 generic.go:356] "Generic (PLEG): container finished" podID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerID="0bf158d9b802ef019be89cdcea2ab57c2233c3516d1df6a048d2eabb7bf16127" exitCode=60 Oct 30 06:00:16.894968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.894929 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerDied","Data":"0bf158d9b802ef019be89cdcea2ab57c2233c3516d1df6a048d2eabb7bf16127"} Oct 30 06:00:16.895280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.895263 5565 scope.go:122] "RemoveContainer" containerID="0bf158d9b802ef019be89cdcea2ab57c2233c3516d1df6a048d2eabb7bf16127" Oct 30 06:00:16.895714 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:16.895693 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm_odh-ai-gateway-infra(7942a8bf-2038-412f-b16d-aa9e1859bf58)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" Oct 30 06:00:16.913056 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:16.913035 5565 scope.go:122] "RemoveContainer" containerID="ae5656e52829432c2c37b1f72f1788d93d6fe14a06bfa70e9420ee1ffead8cab" Oct 30 06:00:17.535149 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:17.535111 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:00:18.079166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.076026 5565 generic.go:356] "Generic (PLEG): container finished" podID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerID="98e72319dcb37923dc8160855b15eb9c0b315002c0c4a7621df282f682318979" exitCode=60 Oct 30 06:00:18.079166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.076112 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" event={"ID":"043a6d14-fd3b-409a-8480-e5e6dd077eb1","Type":"ContainerDied","Data":"98e72319dcb37923dc8160855b15eb9c0b315002c0c4a7621df282f682318979"} Oct 30 06:00:18.079166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.076167 5565 scope.go:122] "RemoveContainer" containerID="ff3cd4708b4ee9b00c904bff0237792866b3d95f93ba157be2581e578f3f5978" Oct 30 06:00:18.079166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.076699 5565 scope.go:122] "RemoveContainer" containerID="98e72319dcb37923dc8160855b15eb9c0b315002c0c4a7621df282f682318979" Oct 30 06:00:18.079166 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:18.077095 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m_odh-ai-gateway-infra(043a6d14-fd3b-409a-8480-e5e6dd077eb1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" Oct 30 06:00:18.124684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.116739 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m"] Oct 30 06:00:18.127595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.127558 5565 generic.go:356] "Generic (PLEG): container finished" podID="ddd289bd-175a-432d-8f42-6eba3a427876" containerID="72eb57dd0fe903be5573c4f7724993f3f6128b3270806028691dd7ad191f23ad" exitCode=60 Oct 30 06:00:18.127721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.127652 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerDied","Data":"72eb57dd0fe903be5573c4f7724993f3f6128b3270806028691dd7ad191f23ad"} Oct 30 06:00:18.146731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.146697 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm"] Oct 30 06:00:18.153671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.153635 5565 scope.go:122] "RemoveContainer" containerID="72eb57dd0fe903be5573c4f7724993f3f6128b3270806028691dd7ad191f23ad" Oct 30 06:00:18.154212 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:18.154159 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c_odh-ai-gateway-infra(ddd289bd-175a-432d-8f42-6eba3a427876)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" Oct 30 06:00:18.213113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.213090 5565 scope.go:122] "RemoveContainer" containerID="fdba2f7446e68b647b858f3b54e277b53f6a777366de3f636c61629658656a6e" Oct 30 06:00:18.459213 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.459187 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:18.511122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.511091 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr\" (UniqueName: \"kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr\") pod \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " Oct 30 06:00:18.511266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.511217 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca\") pod \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\" (UID: \"6b7679c5-5604-4cd1-ada4-ff8abf2c136f\") " Oct 30 06:00:18.512676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.512643 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca" pod "6b7679c5-5604-4cd1-ada4-ff8abf2c136f" (UID: "6b7679c5-5604-4cd1-ada4-ff8abf2c136f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:18.513765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.513735 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr" pod "6b7679c5-5604-4cd1-ada4-ff8abf2c136f" (UID: "6b7679c5-5604-4cd1-ada4-ff8abf2c136f"). InnerVolumeSpecName "kube-api-access-98sxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.513866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.513793 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.516153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.516132 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:18.561337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.561307 5565 scope.go:122] "RemoveContainer" containerID="9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643" Oct 30 06:00:18.562045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.562018 5565 scope.go:122] "RemoveContainer" containerID="63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1" Oct 30 06:00:18.562704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.562290 5565 scope.go:122] "RemoveContainer" containerID="91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30" Oct 30 06:00:18.614679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.614650 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca\") pod \"7942a8bf-2038-412f-b16d-aa9e1859bf58\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " Oct 30 06:00:18.614796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.614713 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf\" (UniqueName: \"kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf\") pod \"7942a8bf-2038-412f-b16d-aa9e1859bf58\" (UID: \"7942a8bf-2038-412f-b16d-aa9e1859bf58\") " Oct 30 06:00:18.615086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.615040 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca" pod "7942a8bf-2038-412f-b16d-aa9e1859bf58" (UID: "7942a8bf-2038-412f-b16d-aa9e1859bf58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:18.618258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.617707 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98sxr\" (UniqueName: \"kubernetes.io/projected/6b7679c5-5604-4cd1-ada4-ff8abf2c136f-kube-api-access-98sxr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.618258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.617730 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7942a8bf-2038-412f-b16d-aa9e1859bf58-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:18.618598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.618518 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf" pod "7942a8bf-2038-412f-b16d-aa9e1859bf58" (UID: "7942a8bf-2038-412f-b16d-aa9e1859bf58"). InnerVolumeSpecName "kube-api-access-tq7vf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:18.719544 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:18.719506 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tq7vf\" (UniqueName: \"kubernetes.io/projected/7942a8bf-2038-412f-b16d-aa9e1859bf58-kube-api-access-tq7vf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.157440 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.157404 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m"] Oct 30 06:00:19.221143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.221074 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" Oct 30 06:00:19.221143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.221102 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m" event={"ID":"6b7679c5-5604-4cd1-ada4-ff8abf2c136f","Type":"ContainerDied","Data":"c5b340bda9051a0a9e167724a2a764d02ee00305ca4e445ea9a1889da7e5af28"} Oct 30 06:00:19.221334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.221169 5565 scope.go:122] "RemoveContainer" containerID="892b3dace8106ac65d529d19d8e3a7ec792f5a2e11380586cb3f7351ff350500" Oct 30 06:00:19.223309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.223272 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerStarted","Data":"1bbb6b5b525b85177c10c02901f041c483cf4e33ae6f8c544d9a734ff907f201"} Oct 30 06:00:19.232145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.231900 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" Oct 30 06:00:19.232671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.232640 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm" event={"ID":"7942a8bf-2038-412f-b16d-aa9e1859bf58","Type":"ContainerDied","Data":"f81568dd7ec51f262d5b93287548940b6e1b794ee7f6d9cde00a25639f67f0d1"} Oct 30 06:00:19.235034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.234996 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerStarted","Data":"c266be6fd7b4cb358396795c7cbd6ce35c6ae7f3d5b9988a59a670db54754641"} Oct 30 06:00:19.243547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.240282 5565 scope.go:122] "RemoveContainer" containerID="0bf158d9b802ef019be89cdcea2ab57c2233c3516d1df6a048d2eabb7bf16127" Oct 30 06:00:19.243547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.242831 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerStarted","Data":"6055c1350a8b31df59f7bdd592c6406c947fd6b9e581982a583fcea1d356e57b"} Oct 30 06:00:19.260938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.260910 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c"] Oct 30 06:00:19.386433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.385772 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m"] Oct 30 06:00:19.392941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.392871 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889000-pbx2m"] Oct 30 06:00:19.533636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.530795 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:19.616906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.614246 5565 scope.go:122] "RemoveContainer" containerID="2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05" Oct 30 06:00:19.621739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.620442 5565 scope.go:122] "RemoveContainer" containerID="e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86" Oct 30 06:00:19.621739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.621678 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7\" (UniqueName: \"kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7\") pod \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " Oct 30 06:00:19.622073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.621862 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca\") pod \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\" (UID: \"043a6d14-fd3b-409a-8480-e5e6dd077eb1\") " Oct 30 06:00:19.623747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.623101 5565 scope.go:122] "RemoveContainer" containerID="de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf" Oct 30 06:00:19.624022 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.623863 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca" pod "043a6d14-fd3b-409a-8480-e5e6dd077eb1" (UID: "043a6d14-fd3b-409a-8480-e5e6dd077eb1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:19.631973 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.626785 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7" pod "043a6d14-fd3b-409a-8480-e5e6dd077eb1" (UID: "043a6d14-fd3b-409a-8480-e5e6dd077eb1"). InnerVolumeSpecName "kube-api-access-bqld7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:19.678284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.678257 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" path="/var/lib/kubelet/pods/6b7679c5-5604-4cd1-ada4-ff8abf2c136f/volumes" Oct 30 06:00:19.678957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.678935 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm"] Oct 30 06:00:19.679052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.678972 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-a14091-29889000-jzngm"] Oct 30 06:00:19.724050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.723988 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqld7\" (UniqueName: \"kubernetes.io/projected/043a6d14-fd3b-409a-8480-e5e6dd077eb1-kube-api-access-bqld7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.724050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.724025 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/043a6d14-fd3b-409a-8480-e5e6dd077eb1-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.803165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.803105 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:19.825288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.825265 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca\") pod \"ddd289bd-175a-432d-8f42-6eba3a427876\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " Oct 30 06:00:19.825396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.825379 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w\" (UniqueName: \"kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w\") pod \"ddd289bd-175a-432d-8f42-6eba3a427876\" (UID: \"ddd289bd-175a-432d-8f42-6eba3a427876\") " Oct 30 06:00:19.825654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.825627 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca" pod "ddd289bd-175a-432d-8f42-6eba3a427876" (UID: "ddd289bd-175a-432d-8f42-6eba3a427876"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:19.826034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.826015 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddd289bd-175a-432d-8f42-6eba3a427876-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:19.827362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.827341 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w" pod "ddd289bd-175a-432d-8f42-6eba3a427876" (UID: "ddd289bd-175a-432d-8f42-6eba3a427876"). InnerVolumeSpecName "kube-api-access-k8g8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:19.927375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:19.927347 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8g8w\" (UniqueName: \"kubernetes.io/projected/ddd289bd-175a-432d-8f42-6eba3a427876-kube-api-access-k8g8w\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:20.254059 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.254025 5565 generic.go:356] "Generic (PLEG): container finished" podID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerID="c266be6fd7b4cb358396795c7cbd6ce35c6ae7f3d5b9988a59a670db54754641" exitCode=60 Oct 30 06:00:20.254654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.254104 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerDied","Data":"c266be6fd7b4cb358396795c7cbd6ce35c6ae7f3d5b9988a59a670db54754641"} Oct 30 06:00:20.254654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.254184 5565 scope.go:122] "RemoveContainer" containerID="9344e1d680e4e98e529ae2ca600bda889844244f03851383526cb24485add643" Oct 30 06:00:20.254654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.254617 5565 scope.go:122] "RemoveContainer" containerID="c266be6fd7b4cb358396795c7cbd6ce35c6ae7f3d5b9988a59a670db54754641" Oct 30 06:00:20.255354 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:20.255317 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv_odh-ai-gateway-infra(542230a4-fb15-4a4d-bc64-d77c8b4d3806)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" Oct 30 06:00:20.257581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.257562 5565 generic.go:356] "Generic (PLEG): container finished" podID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerID="6055c1350a8b31df59f7bdd592c6406c947fd6b9e581982a583fcea1d356e57b" exitCode=60 Oct 30 06:00:20.257680 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.257633 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerDied","Data":"6055c1350a8b31df59f7bdd592c6406c947fd6b9e581982a583fcea1d356e57b"} Oct 30 06:00:20.258005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.257988 5565 scope.go:122] "RemoveContainer" containerID="6055c1350a8b31df59f7bdd592c6406c947fd6b9e581982a583fcea1d356e57b" Oct 30 06:00:20.258290 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:20.258270 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg_odh-ai-gateway-infra(cfc44cce-61f3-4c99-b90a-6edee0789b80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" Oct 30 06:00:20.261110 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.261086 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerStarted","Data":"7aeef76314e658e196bfeae692cff429ea80352efbcf3892df7af172ac634fa2"} Oct 30 06:00:20.263762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.263419 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerID="1bbb6b5b525b85177c10c02901f041c483cf4e33ae6f8c544d9a734ff907f201" exitCode=60 Oct 30 06:00:20.263762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.263587 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerDied","Data":"1bbb6b5b525b85177c10c02901f041c483cf4e33ae6f8c544d9a734ff907f201"} Oct 30 06:00:20.264848 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.264828 5565 scope.go:122] "RemoveContainer" containerID="1bbb6b5b525b85177c10c02901f041c483cf4e33ae6f8c544d9a734ff907f201" Oct 30 06:00:20.265134 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:20.265112 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2_odh-ai-gateway-infra(f5be034e-5537-4b8b-8f9b-6163c306fa87)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" Oct 30 06:00:20.266832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.266352 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" Oct 30 06:00:20.266832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.266358 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m" event={"ID":"043a6d14-fd3b-409a-8480-e5e6dd077eb1","Type":"ContainerDied","Data":"b27f1ade3c33a140e98a02cbb8f8c13e65e43143f7e41e4ac4586ef7fe9bf3fd"} Oct 30 06:00:20.268847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.268826 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" Oct 30 06:00:20.268947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.268828 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c" event={"ID":"ddd289bd-175a-432d-8f42-6eba3a427876","Type":"ContainerDied","Data":"93fb25cda9e55ce44edd205eec35a870f626bc01f13c739463655d494f69c65d"} Oct 30 06:00:20.275244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.275223 5565 scope.go:122] "RemoveContainer" containerID="63fa47615378308467736d60a8d0cf2ab4eabff17b04d463528186ce0d1be8f1" Oct 30 06:00:20.288913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.288897 5565 scope.go:122] "RemoveContainer" containerID="91274ab26196f456818e7c30d3f2cc6945dc6444ea1d5cd036db214889034a30" Oct 30 06:00:20.303757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.303736 5565 scope.go:122] "RemoveContainer" containerID="98e72319dcb37923dc8160855b15eb9c0b315002c0c4a7621df282f682318979" Oct 30 06:00:20.313094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.313077 5565 scope.go:122] "RemoveContainer" containerID="72eb57dd0fe903be5573c4f7724993f3f6128b3270806028691dd7ad191f23ad" Oct 30 06:00:20.445494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.445418 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv"] Oct 30 06:00:20.482016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.481990 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m"] Oct 30 06:00:20.490813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.490766 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889000-hbg7m"] Oct 30 06:00:20.500711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.500679 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2"] Oct 30 06:00:20.525836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.525805 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c"] Oct 30 06:00:20.537753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.537726 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-1d8c2f-29889000-2ck5c"] Oct 30 06:00:20.705585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:20.705504 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg"] Oct 30 06:00:21.289160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.289128 5565 generic.go:356] "Generic (PLEG): container finished" podID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerID="dac29680b1ff02bba04bc7abe972adecb80c0f7aefa99d5463cadd54f9a3ca69" exitCode=60 Oct 30 06:00:21.289547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.289213 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerDied","Data":"dac29680b1ff02bba04bc7abe972adecb80c0f7aefa99d5463cadd54f9a3ca69"} Oct 30 06:00:21.289547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.289311 5565 scope.go:122] "RemoveContainer" containerID="e0abc3d9fd0f4788f024126aa2df53bf4b5b06e2a6e7ff9defbc92e5aeb5fc86" Oct 30 06:00:21.289782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.289762 5565 scope.go:122] "RemoveContainer" containerID="dac29680b1ff02bba04bc7abe972adecb80c0f7aefa99d5463cadd54f9a3ca69" Oct 30 06:00:21.290234 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:21.290199 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx_odh-ai-gateway-infra(04d1cb96-7416-41e9-8a1b-61544771a98b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" Oct 30 06:00:21.291487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.291468 5565 generic.go:356] "Generic (PLEG): container finished" podID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerID="7aeef76314e658e196bfeae692cff429ea80352efbcf3892df7af172ac634fa2" exitCode=60 Oct 30 06:00:21.291583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.291564 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerDied","Data":"7aeef76314e658e196bfeae692cff429ea80352efbcf3892df7af172ac634fa2"} Oct 30 06:00:21.291964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.291948 5565 scope.go:122] "RemoveContainer" containerID="7aeef76314e658e196bfeae692cff429ea80352efbcf3892df7af172ac634fa2" Oct 30 06:00:21.292368 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:21.292327 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q_odh-ai-gateway-infra(940e5c05-e3c9-4074-bab3-21d1771bbdcc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" Oct 30 06:00:21.296001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.295975 5565 generic.go:356] "Generic (PLEG): container finished" podID="7436eeec-3732-4338-9c91-373887957bde" containerID="fcb65885b6d455c1f00c56484c1c203b175589fe8bf17c8d78be350c526362d7" exitCode=60 Oct 30 06:00:21.296093 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.296015 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerDied","Data":"fcb65885b6d455c1f00c56484c1c203b175589fe8bf17c8d78be350c526362d7"} Oct 30 06:00:21.296484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.296465 5565 scope.go:122] "RemoveContainer" containerID="fcb65885b6d455c1f00c56484c1c203b175589fe8bf17c8d78be350c526362d7" Oct 30 06:00:21.296897 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:21.296874 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889000-rzrtp_odh-ai-gateway-infra(7436eeec-3732-4338-9c91-373887957bde)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" podUID="7436eeec-3732-4338-9c91-373887957bde" Oct 30 06:00:21.307455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.307417 5565 scope.go:122] "RemoveContainer" containerID="2fcb62b774f673489887467fbd61df640f571c2bff9ceaeb63eb39cab0867b05" Oct 30 06:00:21.329864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.329840 5565 scope.go:122] "RemoveContainer" containerID="de55dfd0929c374e0ade7484dd71e95a754d414931eda9f5b27b07b995c9afdf" Oct 30 06:00:21.369336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.368876 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q"] Oct 30 06:00:21.506445 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.506359 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:21.522247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.522219 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:21.567961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.567937 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:21.568133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.568111 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca\") pod \"f5be034e-5537-4b8b-8f9b-6163c306fa87\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " Oct 30 06:00:21.568289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.568274 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca\") pod \"cfc44cce-61f3-4c99-b90a-6edee0789b80\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " Oct 30 06:00:21.568378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.568358 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m\" (UniqueName: \"kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m\") pod \"cfc44cce-61f3-4c99-b90a-6edee0789b80\" (UID: \"cfc44cce-61f3-4c99-b90a-6edee0789b80\") " Oct 30 06:00:21.568468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.568440 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca" pod "f5be034e-5537-4b8b-8f9b-6163c306fa87" (UID: "f5be034e-5537-4b8b-8f9b-6163c306fa87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:21.568468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.568459 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2\" (UniqueName: \"kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2\") pod \"f5be034e-5537-4b8b-8f9b-6163c306fa87\" (UID: \"f5be034e-5537-4b8b-8f9b-6163c306fa87\") " Oct 30 06:00:21.568827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.568796 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca" pod "cfc44cce-61f3-4c99-b90a-6edee0789b80" (UID: "cfc44cce-61f3-4c99-b90a-6edee0789b80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:21.569352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.569328 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f5be034e-5537-4b8b-8f9b-6163c306fa87-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.569440 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.569357 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfc44cce-61f3-4c99-b90a-6edee0789b80-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.570796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.570767 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m" pod "cfc44cce-61f3-4c99-b90a-6edee0789b80" (UID: "cfc44cce-61f3-4c99-b90a-6edee0789b80"). InnerVolumeSpecName "kube-api-access-r7m7m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:21.570956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.570936 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2" pod "f5be034e-5537-4b8b-8f9b-6163c306fa87" (UID: "f5be034e-5537-4b8b-8f9b-6163c306fa87"). InnerVolumeSpecName "kube-api-access-tt5t2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:21.574962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.574935 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" path="/var/lib/kubelet/pods/043a6d14-fd3b-409a-8480-e5e6dd077eb1/volumes" Oct 30 06:00:21.575460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.575439 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" path="/var/lib/kubelet/pods/7942a8bf-2038-412f-b16d-aa9e1859bf58/volumes" Oct 30 06:00:21.575949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.575929 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" path="/var/lib/kubelet/pods/ddd289bd-175a-432d-8f42-6eba3a427876/volumes" Oct 30 06:00:21.670605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.670574 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca\") pod \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " Oct 30 06:00:21.670798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.670723 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6\" (UniqueName: \"kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6\") pod \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\" (UID: \"542230a4-fb15-4a4d-bc64-d77c8b4d3806\") " Oct 30 06:00:21.671018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.670988 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca" pod "542230a4-fb15-4a4d-bc64-d77c8b4d3806" (UID: "542230a4-fb15-4a4d-bc64-d77c8b4d3806"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:21.671565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.671517 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7m7m\" (UniqueName: \"kubernetes.io/projected/cfc44cce-61f3-4c99-b90a-6edee0789b80-kube-api-access-r7m7m\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.671648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.671568 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt5t2\" (UniqueName: \"kubernetes.io/projected/f5be034e-5537-4b8b-8f9b-6163c306fa87-kube-api-access-tt5t2\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.671648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.671585 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/542230a4-fb15-4a4d-bc64-d77c8b4d3806-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:21.672922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.672895 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6" pod "542230a4-fb15-4a4d-bc64-d77c8b4d3806" (UID: "542230a4-fb15-4a4d-bc64-d77c8b4d3806"). InnerVolumeSpecName "kube-api-access-5dcn6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:21.773256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:21.772948 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dcn6\" (UniqueName: \"kubernetes.io/projected/542230a4-fb15-4a4d-bc64-d77c8b4d3806-kube-api-access-5dcn6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.310170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.310135 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" event={"ID":"f5be034e-5537-4b8b-8f9b-6163c306fa87","Type":"ContainerDied","Data":"949cf3918c3e560dbae4b7438c06ef3b23701858905d656e1cc2bd3cf9ff4279"} Oct 30 06:00:22.310602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.310207 5565 scope.go:122] "RemoveContainer" containerID="1bbb6b5b525b85177c10c02901f041c483cf4e33ae6f8c544d9a734ff907f201" Oct 30 06:00:22.310602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.310143 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2" Oct 30 06:00:22.320814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.318285 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" event={"ID":"542230a4-fb15-4a4d-bc64-d77c8b4d3806","Type":"ContainerDied","Data":"b03ad62b71f323e48d345f804034d32ff5a1f83436264a3c32639729b70b9b56"} Oct 30 06:00:22.320814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.318391 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv" Oct 30 06:00:22.321867 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.321839 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" event={"ID":"cfc44cce-61f3-4c99-b90a-6edee0789b80","Type":"ContainerDied","Data":"1da28346e9efa93ebc8983a8b98f9c43d8efa41e3856b1936147b7caccad5f2a"} Oct 30 06:00:22.321982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.321888 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg" Oct 30 06:00:22.329635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.329613 5565 scope.go:122] "RemoveContainer" containerID="c266be6fd7b4cb358396795c7cbd6ce35c6ae7f3d5b9988a59a670db54754641" Oct 30 06:00:22.344075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.344052 5565 scope.go:122] "RemoveContainer" containerID="6055c1350a8b31df59f7bdd592c6406c947fd6b9e581982a583fcea1d356e57b" Oct 30 06:00:22.411098 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.411062 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv"] Oct 30 06:00:22.418339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.418305 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-0b481b-29889000-nj7nv"] Oct 30 06:00:22.449468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.449428 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg"] Oct 30 06:00:22.455864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.455836 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-086c60-29889000-555mg"] Oct 30 06:00:22.462012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.461986 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx"] Oct 30 06:00:22.494275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.494242 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2"] Oct 30 06:00:22.505907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.505822 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-3abe42-29889000-s7ml2"] Oct 30 06:00:22.519057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.519006 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:22.584429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.584350 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca\") pod \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " Oct 30 06:00:22.584601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.584452 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4\" (UniqueName: \"kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4\") pod \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\" (UID: \"940e5c05-e3c9-4074-bab3-21d1771bbdcc\") " Oct 30 06:00:22.584823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.584795 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca" pod "940e5c05-e3c9-4074-bab3-21d1771bbdcc" (UID: "940e5c05-e3c9-4074-bab3-21d1771bbdcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:22.585164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.585138 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/940e5c05-e3c9-4074-bab3-21d1771bbdcc-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.591598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.587013 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4" pod "940e5c05-e3c9-4074-bab3-21d1771bbdcc" (UID: "940e5c05-e3c9-4074-bab3-21d1771bbdcc"). InnerVolumeSpecName "kube-api-access-n2bc4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:22.611430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.611016 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp"] Oct 30 06:00:22.671013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.670984 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:22.716659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.709365 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx\") pod \"04d1cb96-7416-41e9-8a1b-61544771a98b\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " Oct 30 06:00:22.716659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.715431 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca\") pod \"04d1cb96-7416-41e9-8a1b-61544771a98b\" (UID: \"04d1cb96-7416-41e9-8a1b-61544771a98b\") " Oct 30 06:00:22.717442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.716999 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca" pod "04d1cb96-7416-41e9-8a1b-61544771a98b" (UID: "04d1cb96-7416-41e9-8a1b-61544771a98b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:22.717442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.717059 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2bc4\" (UniqueName: \"kubernetes.io/projected/940e5c05-e3c9-4074-bab3-21d1771bbdcc-kube-api-access-n2bc4\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.742748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.729726 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx" pod "04d1cb96-7416-41e9-8a1b-61544771a98b" (UID: "04d1cb96-7416-41e9-8a1b-61544771a98b"). InnerVolumeSpecName "kube-api-access-gplkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:22.819520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.819480 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gplkx\" (UniqueName: \"kubernetes.io/projected/04d1cb96-7416-41e9-8a1b-61544771a98b-kube-api-access-gplkx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.819520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.819521 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04d1cb96-7416-41e9-8a1b-61544771a98b-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:22.870829 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.870807 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:22.920945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.920847 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca\") pod \"7436eeec-3732-4338-9c91-373887957bde\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " Oct 30 06:00:22.920945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.920897 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9\" (UniqueName: \"kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9\") pod \"7436eeec-3732-4338-9c91-373887957bde\" (UID: \"7436eeec-3732-4338-9c91-373887957bde\") " Oct 30 06:00:22.922194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.922152 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca" pod "7436eeec-3732-4338-9c91-373887957bde" (UID: "7436eeec-3732-4338-9c91-373887957bde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:22.924062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:22.924027 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9" pod "7436eeec-3732-4338-9c91-373887957bde" (UID: "7436eeec-3732-4338-9c91-373887957bde"). InnerVolumeSpecName "kube-api-access-pxkr9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:23.022925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.022894 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7436eeec-3732-4338-9c91-373887957bde-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:23.022925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.022924 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxkr9\" (UniqueName: \"kubernetes.io/projected/7436eeec-3732-4338-9c91-373887957bde-kube-api-access-pxkr9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:23.330930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.330902 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" Oct 30 06:00:23.331395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.330901 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx" event={"ID":"04d1cb96-7416-41e9-8a1b-61544771a98b","Type":"ContainerDied","Data":"c4f033abe6a943a51872d9aba3f3d324b76be8d44d4148433c6a9692c3c23c2c"} Oct 30 06:00:23.331395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.331063 5565 scope.go:122] "RemoveContainer" containerID="dac29680b1ff02bba04bc7abe972adecb80c0f7aefa99d5463cadd54f9a3ca69" Oct 30 06:00:23.332573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.332518 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" event={"ID":"940e5c05-e3c9-4074-bab3-21d1771bbdcc","Type":"ContainerDied","Data":"1cc3fd5f52e1ae8db432611f422408728e7cc850ed2901e09757f46152d7526c"} Oct 30 06:00:23.332669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.332638 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q" Oct 30 06:00:23.336419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.336393 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" Oct 30 06:00:23.336705 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.336445 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp" event={"ID":"7436eeec-3732-4338-9c91-373887957bde","Type":"ContainerDied","Data":"86ee8dd9f0522747d4d3d97b48ca004085cdf68a091b4053c1c2df30ca7bb571"} Oct 30 06:00:23.351722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.345271 5565 scope.go:122] "RemoveContainer" containerID="7aeef76314e658e196bfeae692cff429ea80352efbcf3892df7af172ac634fa2" Oct 30 06:00:23.360439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.360415 5565 scope.go:122] "RemoveContainer" containerID="fcb65885b6d455c1f00c56484c1c203b175589fe8bf17c8d78be350c526362d7" Oct 30 06:00:23.423752 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.423723 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx"] Oct 30 06:00:23.427464 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.427438 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889000-56vpx"] Oct 30 06:00:23.463954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.463932 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q"] Oct 30 06:00:23.470670 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.470647 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-0382c2-29889000-vr25q"] Oct 30 06:00:23.509870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.509847 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp"] Oct 30 06:00:23.514904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.514884 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889000-rzrtp"] Oct 30 06:00:23.569568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.569546 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" path="/var/lib/kubelet/pods/04d1cb96-7416-41e9-8a1b-61544771a98b/volumes" Oct 30 06:00:23.570029 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.570016 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" path="/var/lib/kubelet/pods/542230a4-fb15-4a4d-bc64-d77c8b4d3806/volumes" Oct 30 06:00:23.570460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.570445 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7436eeec-3732-4338-9c91-373887957bde" path="/var/lib/kubelet/pods/7436eeec-3732-4338-9c91-373887957bde/volumes" Oct 30 06:00:23.571192 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.571177 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" path="/var/lib/kubelet/pods/940e5c05-e3c9-4074-bab3-21d1771bbdcc/volumes" Oct 30 06:00:23.571594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.571581 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" path="/var/lib/kubelet/pods/cfc44cce-61f3-4c99-b90a-6edee0789b80/volumes" Oct 30 06:00:23.571981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:23.571969 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" path="/var/lib/kubelet/pods/f5be034e-5537-4b8b-8f9b-6163c306fa87/volumes" Oct 30 06:00:30.346552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.346011 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p"] Oct 30 06:00:30.347188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347159 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.347188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347187 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347206 5565 cpu_manager.go:414] "containerMap: removing container" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347214 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347230 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347238 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347254 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347260 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347273 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347280 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347290 5565 cpu_manager.go:414] "containerMap: removing container" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347297 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347309 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347316 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347327 5565 cpu_manager.go:414] "containerMap: removing container" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.347334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347333 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347349 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347356 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347376 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347383 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347397 5565 cpu_manager.go:414] "containerMap: removing container" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347404 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347413 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347420 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347447 5565 cpu_manager.go:414] "containerMap: removing container" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347455 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347466 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347473 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347496 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347502 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347515 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347522 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347549 5565 cpu_manager.go:414] "containerMap: removing container" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347556 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347565 5565 cpu_manager.go:414] "containerMap: removing container" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347573 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347586 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347594 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347608 5565 cpu_manager.go:414] "containerMap: removing container" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347616 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347628 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347635 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347659 5565 cpu_manager.go:414] "containerMap: removing container" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347666 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347849 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347863 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347873 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347882 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347892 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347900 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347908 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347917 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347929 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347940 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347948 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347960 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347972 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347980 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347988 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.347995 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.348005 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.348014 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.348023 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.348034 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.348049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.348043 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.352607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.352578 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.401890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.401860 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p"] Oct 30 06:00:30.451061 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.451022 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt"] Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452174 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452201 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452218 5565 cpu_manager.go:414] "containerMap: removing container" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452227 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452249 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452257 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7942a8bf-2038-412f-b16d-aa9e1859bf58" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452272 5565 cpu_manager.go:414] "containerMap: removing container" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452279 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452294 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452302 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452329 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452346 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452362 5565 cpu_manager.go:414] "containerMap: removing container" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452369 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452378 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452384 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452592 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfc44cce-61f3-4c99-b90a-6edee0789b80" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452618 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="043a6d14-fd3b-409a-8480-e5e6dd077eb1" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452630 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7436eeec-3732-4338-9c91-373887957bde" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452643 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddd289bd-175a-432d-8f42-6eba3a427876" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452653 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5be034e-5537-4b8b-8f9b-6163c306fa87" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452671 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="940e5c05-e3c9-4074-bab3-21d1771bbdcc" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452682 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="542230a4-fb15-4a4d-bc64-d77c8b4d3806" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452696 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b7679c5-5604-4cd1-ada4-ff8abf2c136f" containerName="cleanup" Oct 30 06:00:30.452785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.452708 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="04d1cb96-7416-41e9-8a1b-61544771a98b" containerName="cleanup" Oct 30 06:00:30.456094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456067 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgkbt\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456233 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456321 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456304 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456332 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456448 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456501 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456572 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456644 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.456731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.456701 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.457834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.457811 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.494879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.494848 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt"] Oct 30 06:00:30.557968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.557940 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.557977 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558051 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558161 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558193 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558306 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558260 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558306 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558302 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558380 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558428 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgkbt\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558480 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558581 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558634 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558681 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558762 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2fhk\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558848 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558802 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.558848 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558805 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.558955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558872 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.559014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558953 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.559014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558972 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.559014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.558991 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.559178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.559025 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.559178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.559084 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.559397 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.559379 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.561239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.561219 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.561483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.561465 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.587707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.587682 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.588813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.588789 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgkbt\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.660607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.660542 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2fhk\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.660607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.660581 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.660607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.660607 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.660837 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.660812 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.660963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.660947 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.660990 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661058 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661122 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661145 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661302 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661313 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661402 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661756 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.661860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.661833 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.663298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.663279 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.663373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.663312 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.677575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.677549 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:30.685685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.685663 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2fhk\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.690964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.690943 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token\") pod \"e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:30.773970 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:30.773939 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:31.001088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.001059 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p"] Oct 30 06:00:31.001493 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:31.001471 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e6953c_e2a4_4f78_9050_58b08787366b.slice/crio-f899e086f2d4ff07f078ac00627673f235a9528bb493989397493d75acbeeb1a WatchSource:0}: Error finding container f899e086f2d4ff07f078ac00627673f235a9528bb493989397493d75acbeeb1a: Status 404 returned error can't find the container with id f899e086f2d4ff07f078ac00627673f235a9528bb493989397493d75acbeeb1a Oct 30 06:00:31.004247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.004216 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:31.004330 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.004279 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:31.004330 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.004309 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:31.093676 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:31.093653 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d588f4_8fe6_417c_8e61_7b644f8190a3.slice/crio-89db2acbfa1980ec7ac8f5c3c93e043c2c334a4229d53d1027c5258f29cbd7c0 WatchSource:0}: Error finding container 89db2acbfa1980ec7ac8f5c3c93e043c2c334a4229d53d1027c5258f29cbd7c0: Status 404 returned error can't find the container with id 89db2acbfa1980ec7ac8f5c3c93e043c2c334a4229d53d1027c5258f29cbd7c0 Oct 30 06:00:31.094281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.094257 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt"] Oct 30 06:00:31.096831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.096801 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:31.096924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.096861 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:31.096924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.096893 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:00:31.445600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.445563 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" event={"ID":"b9e6953c-e2a4-4f78-9050-58b08787366b","Type":"ContainerStarted","Data":"2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238"} Oct 30 06:00:31.446146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.445621 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" event={"ID":"b9e6953c-e2a4-4f78-9050-58b08787366b","Type":"ContainerStarted","Data":"f899e086f2d4ff07f078ac00627673f235a9528bb493989397493d75acbeeb1a"} Oct 30 06:00:31.447666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.447625 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" event={"ID":"90d588f4-8fe6-417c-8e61-7b644f8190a3","Type":"ContainerStarted","Data":"42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2"} Oct 30 06:00:31.447816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.447689 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" event={"ID":"90d588f4-8fe6-417c-8e61-7b644f8190a3","Type":"ContainerStarted","Data":"89db2acbfa1980ec7ac8f5c3c93e043c2c334a4229d53d1027c5258f29cbd7c0"} Oct 30 06:00:31.497430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.497350 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podStartSLOduration=1.497332434 podStartE2EDuration="1.497332434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:31.494738465 +0000 UTC m=+1738.070425533" watchObservedRunningTime="2026-10-30 06:00:31.497332434 +0000 UTC m=+1738.073019489" Oct 30 06:00:31.552275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.552177 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podStartSLOduration=1.55216324 podStartE2EDuration="1.55216324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:31.54991693 +0000 UTC m=+1738.125604001" watchObservedRunningTime="2026-10-30 06:00:31.55216324 +0000 UTC m=+1738.127850294" Oct 30 06:00:31.677799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.677763 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:31.774226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:31.774187 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:32.686752 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:32.683419 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:00:32.686752 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:32.683485 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 06:00:32.777409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:32.777374 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:00:32.777719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:32.777689 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 06:00:33.719971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:33.719934 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:00:33.720588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:33.720554 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 06:00:33.731923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:33.731893 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:00:33.777551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:33.776927 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:00:33.777551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:33.776984 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 06:00:34.682553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:34.681001 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:00:34.682553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:34.681056 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 06:00:34.779549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:34.778868 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:00:34.779549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:34.778926 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 06:00:35.735557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:35.688944 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:00:35.735557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:35.689009 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 06:00:35.825586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:35.781209 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:00:35.825586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:35.781269 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 06:00:36.683301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:36.683266 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body= Oct 30 06:00:36.683610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:36.683578 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" Oct 30 06:00:36.774906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:36.774864 5565 patch_prober.go:29] interesting pod/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body= Oct 30 06:00:36.776292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:36.776260 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" Oct 30 06:00:37.418201 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.418171 5565 scope.go:122] "RemoveContainer" containerID="b922ae2d719c71b262c2ba8487274857e55f904ce2cc0b27f96e764a9b1a9f4f" Oct 30 06:00:37.731718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.731624 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:37.732290 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.732265 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:37.733381 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.733360 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:00:37.802938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.802893 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:37.804592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.804569 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:37.806151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:37.806127 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:00:46.034980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:46.034930 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5"] Oct 30 06:00:46.048639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:46.048613 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a49fcf92-m5fx5"] Oct 30 06:00:47.535586 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:47.535548 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:00:47.570929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:47.570890 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11f60d19-e41b-45d0-b527-7d87ad79f416" path="/var/lib/kubelet/pods/11f60d19-e41b-45d0-b527-7d87ad79f416/volumes" Oct 30 06:00:49.381387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:49.381354 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748"] Oct 30 06:00:49.381826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:49.381575 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" containerID="cri-o://a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9" gracePeriod=30 Oct 30 06:00:54.673740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.673703 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 06:00:54.788645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.788610 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.788828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.788711 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.788828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.788820 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.788932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.788859 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.788932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.788905 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:54.789038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789014 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.789113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789080 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.789164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789128 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:54.789217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789177 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.789519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789459 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:54.789519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789466 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:00:54.789701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789600 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.789701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789643 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token\") pod \"072249aa-e351-49d2-81d0-e20398bf9c15\" (UID: \"072249aa-e351-49d2-81d0-e20398bf9c15\") " Oct 30 06:00:54.789878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.789846 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:54.790564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.790543 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.790739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.790726 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.790848 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.790834 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.790948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.790934 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/072249aa-e351-49d2-81d0-e20398bf9c15-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.791042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.791031 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.791876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.791839 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:00:54.792477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.792439 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "kube-api-access-zrfj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:54.793075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.793052 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:00:54.793894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.793865 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token" pod "072249aa-e351-49d2-81d0-e20398bf9c15" (UID: "072249aa-e351-49d2-81d0-e20398bf9c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:00:54.893476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.893436 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/072249aa-e351-49d2-81d0-e20398bf9c15-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.893476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.893476 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.893710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.893515 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfj9\" (UniqueName: \"kubernetes.io/projected/072249aa-e351-49d2-81d0-e20398bf9c15-kube-api-access-zrfj9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:54.893710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:54.893544 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/072249aa-e351-49d2-81d0-e20398bf9c15-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:00:55.025829 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.025785 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j"] Oct 30 06:00:55.026763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.026738 5565 cpu_manager.go:414] "containerMap: removing container" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" Oct 30 06:00:55.026763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.026762 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" Oct 30 06:00:55.027020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.027005 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" containerName="istio-proxy" Oct 30 06:00:55.031118 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.031095 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.058428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.056767 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-8de592-serving-cert\"" Oct 30 06:00:55.063987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.063954 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j"] Oct 30 06:00:55.069575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.069502 5565 generic.go:356] "Generic (PLEG): container finished" podID="072249aa-e351-49d2-81d0-e20398bf9c15" containerID="a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9" exitCode=0 Oct 30 06:00:55.069575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.069565 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" event={"ID":"072249aa-e351-49d2-81d0-e20398bf9c15","Type":"ContainerDied","Data":"a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9"} Oct 30 06:00:55.069813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.069585 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" Oct 30 06:00:55.069813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.069626 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748" event={"ID":"072249aa-e351-49d2-81d0-e20398bf9c15","Type":"ContainerDied","Data":"948eb766c1c5a0562b4983ab2b1fe61bd2e65bde2c797b48b040bbad45af541e"} Oct 30 06:00:55.069813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.069654 5565 scope.go:122] "RemoveContainer" containerID="a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9" Oct 30 06:00:55.087909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.087883 5565 scope.go:122] "RemoveContainer" containerID="a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9" Oct 30 06:00:55.089184 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:00:55.089147 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9\": container with ID starting with a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9 not found: ID does not exist" containerID="a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9" Oct 30 06:00:55.089330 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.089193 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9"} err="failed to get container status \"a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9\": rpc error: code = NotFound desc = could not find container \"a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9\": container with ID starting with a777d2632c697ff51702d6d17bbc4aedd8db74380ae7e7de2cc84a694631e9e9 not found: ID does not exist" Oct 30 06:00:55.096542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.096496 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.096726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.096680 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.096891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.096867 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sxgg\" (UniqueName: \"kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.097012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.096982 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.134989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.134946 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748"] Oct 30 06:00:55.141109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.141075 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-pv748"] Oct 30 06:00:55.211580 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.210498 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.213038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.212988 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.213420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.213396 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.213724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.213704 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sxgg\" (UniqueName: \"kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.218015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.217989 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.218814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.218790 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.252097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.251868 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.270507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.270471 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sxgg\" (UniqueName: \"kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg\") pod \"maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.348007 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.347910 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:00:55.569808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.569776 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="072249aa-e351-49d2-81d0-e20398bf9c15" path="/var/lib/kubelet/pods/072249aa-e351-49d2-81d0-e20398bf9c15/volumes" Oct 30 06:00:55.666393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.666360 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j"] Oct 30 06:00:55.667906 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:00:55.667869 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7de6c1_a8ab_4b17_ab4f_e03638bb7f9c.slice/crio-fed6c5d513a640bb8acfea8f189d92f7e515a2c45028143cfd2499ca4a59df3e WatchSource:0}: Error finding container fed6c5d513a640bb8acfea8f189d92f7e515a2c45028143cfd2499ca4a59df3e: Status 404 returned error can't find the container with id fed6c5d513a640bb8acfea8f189d92f7e515a2c45028143cfd2499ca4a59df3e Oct 30 06:00:55.669462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:55.669440 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:00:56.089376 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:56.089336 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" event={"ID":"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c","Type":"ContainerStarted","Data":"fed6c5d513a640bb8acfea8f189d92f7e515a2c45028143cfd2499ca4a59df3e"} Oct 30 06:00:57.101624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:57.101577 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" event={"ID":"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c","Type":"ContainerStarted","Data":"bdfdd98d246134d6d151dbd13191bdb96708a29561cfcdf92957cf25ac29d130"} Oct 30 06:00:57.102158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:00:57.101665 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:01:00.203548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.203484 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" podStartSLOduration=5.795994017 podStartE2EDuration="6.203468896s" totalImagesPullingTime="407.474879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:00:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:00:57.176374811 +0000 UTC m=+1763.752061866" watchObservedRunningTime="2026-10-30 06:01:00.203468896 +0000 UTC m=+1766.779155947" Oct 30 06:01:00.206417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.206391 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd"] Oct 30 06:01:00.211930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.211905 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.231257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.231231 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd"] Oct 30 06:01:00.237193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.237164 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.237564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.237497 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhkn\" (UniqueName: \"kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.237690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.237664 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.240294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.240272 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:01:00.241157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.241138 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:01:00.339783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.339749 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.339989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.339935 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhkn\" (UniqueName: \"kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.340060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.340026 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.340849 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.340818 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.342326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.342295 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.365253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.365224 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhkn\" (UniqueName: \"kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn\") pod \"maas-api-revoke-sub-bc288054-g57gd\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.534010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.528600 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:00.878487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:00.878456 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd"] Oct 30 06:01:00.878641 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:01:00.878510 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5546c62_7dcd_47a4_a3f4_9fa1ca646b04.slice/crio-0f4b3eb9b19960a18a9a350d9849106a1a24387d2b82baa1cfffeeb29b2461c6 WatchSource:0}: Error finding container 0f4b3eb9b19960a18a9a350d9849106a1a24387d2b82baa1cfffeeb29b2461c6: Status 404 returned error can't find the container with id 0f4b3eb9b19960a18a9a350d9849106a1a24387d2b82baa1cfffeeb29b2461c6 Oct 30 06:01:01.135152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:01.135077 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" event={"ID":"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04","Type":"ContainerStarted","Data":"0f4b3eb9b19960a18a9a350d9849106a1a24387d2b82baa1cfffeeb29b2461c6"} Oct 30 06:01:02.148633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:02.148251 5565 generic.go:356] "Generic (PLEG): container finished" podID="b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" containerID="84362b4297f954e13b94994fb2547b0534a2c64ff556ec7e0e787171b2552fe8" exitCode=0 Oct 30 06:01:02.148633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:02.148394 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" event={"ID":"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04","Type":"ContainerDied","Data":"84362b4297f954e13b94994fb2547b0534a2c64ff556ec7e0e787171b2552fe8"} Oct 30 06:01:03.115742 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:03.115709 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:01:04.126558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.122724 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:04.325221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.284084 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca\") pod \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " Oct 30 06:01:04.325221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.284269 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn\" (UniqueName: \"kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn\") pod \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " Oct 30 06:01:04.325221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.293215 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca\") pod \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\" (UID: \"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04\") " Oct 30 06:01:04.325221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.312679 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:01:04.363610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.362712 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 06:01:04.400146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.399894 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca" pod "b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" (UID: "b5546c62-7dcd-47a4-a3f4-9fa1ca646b04"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:01:04.404045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.404004 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" event={"ID":"b5546c62-7dcd-47a4-a3f4-9fa1ca646b04","Type":"ContainerDied","Data":"0f4b3eb9b19960a18a9a350d9849106a1a24387d2b82baa1cfffeeb29b2461c6"} Oct 30 06:01:04.404303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.404286 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f4b3eb9b19960a18a9a350d9849106a1a24387d2b82baa1cfffeeb29b2461c6" Oct 30 06:01:04.404647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.404628 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd" Oct 30 06:01:04.408373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.408329 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:04.438271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.434770 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca" pod "b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" (UID: "b5546c62-7dcd-47a4-a3f4-9fa1ca646b04"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:01:04.438271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.434995 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn" pod "b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" (UID: "b5546c62-7dcd-47a4-a3f4-9fa1ca646b04"). InnerVolumeSpecName "kube-api-access-tzhkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:04.510308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.510254 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhkn\" (UniqueName: \"kubernetes.io/projected/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-kube-api-access-tzhkn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:04.510308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:04.510300 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:14.815606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.815563 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7"] Oct 30 06:01:14.816415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.816397 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" containerName="revoke-keys" Oct 30 06:01:14.816415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.816415 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" containerName="revoke-keys" Oct 30 06:01:14.816564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.816552 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" containerName="revoke-keys" Oct 30 06:01:14.820852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.820828 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:14.841877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.841842 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7"] Oct 30 06:01:14.861256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.861225 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:01:14.861409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.861300 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:01:14.935190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.935150 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnlc5\" (UniqueName: \"kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:14.935391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.935348 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:14.935686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:14.935664 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.037498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.037424 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.037498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.037523 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnlc5\" (UniqueName: \"kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.037845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.037630 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.038268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.038238 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.040194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.040169 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.062968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.062928 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnlc5\" (UniqueName: \"kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5\") pod \"maas-api-revoke-sub-2c91a764-dvcj7\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.135907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.135799 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:15.455698 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:01:15.455649 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d4dc6c_a13f_419f_9df2_977e50e6e452.slice/crio-ab0b9485e749fd4ab4515f260343d323c90f9f3a176105a0048a0b919be7af6c WatchSource:0}: Error finding container ab0b9485e749fd4ab4515f260343d323c90f9f3a176105a0048a0b919be7af6c: Status 404 returned error can't find the container with id ab0b9485e749fd4ab4515f260343d323c90f9f3a176105a0048a0b919be7af6c Oct 30 06:01:15.455941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.455914 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7"] Oct 30 06:01:15.512761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:15.512720 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" event={"ID":"d0d4dc6c-a13f-419f-9df2-977e50e6e452","Type":"ContainerStarted","Data":"ab0b9485e749fd4ab4515f260343d323c90f9f3a176105a0048a0b919be7af6c"} Oct 30 06:01:16.528780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:16.528732 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" event={"ID":"d0d4dc6c-a13f-419f-9df2-977e50e6e452","Type":"ContainerStarted","Data":"b061a779c544ba32c6ded98091e3b0d612c55902f26fad261fe92dc483c1a18a"} Oct 30 06:01:16.579823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:16.579611 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" podStartSLOduration=2.579589942 podStartE2EDuration="2.579589942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:16.574851753 +0000 UTC m=+1783.150538808" watchObservedRunningTime="2026-10-30 06:01:16.579589942 +0000 UTC m=+1783.155276998" Oct 30 06:01:17.540936 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:01:17.540891 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:01:17.542814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:17.542784 5565 generic.go:356] "Generic (PLEG): container finished" podID="d0d4dc6c-a13f-419f-9df2-977e50e6e452" containerID="b061a779c544ba32c6ded98091e3b0d612c55902f26fad261fe92dc483c1a18a" exitCode=0 Oct 30 06:01:17.543065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:17.542867 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" event={"ID":"d0d4dc6c-a13f-419f-9df2-977e50e6e452","Type":"ContainerDied","Data":"b061a779c544ba32c6ded98091e3b0d612c55902f26fad261fe92dc483c1a18a"} Oct 30 06:01:18.808098 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.808070 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:18.820448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.820414 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca\") pod \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " Oct 30 06:01:18.820652 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.820637 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5\" (UniqueName: \"kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5\") pod \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " Oct 30 06:01:18.820705 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.820676 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca\") pod \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\" (UID: \"d0d4dc6c-a13f-419f-9df2-977e50e6e452\") " Oct 30 06:01:18.821070 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.821035 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca" pod "d0d4dc6c-a13f-419f-9df2-977e50e6e452" (UID: "d0d4dc6c-a13f-419f-9df2-977e50e6e452"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:01:18.821382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.821366 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:18.822611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.822580 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca" pod "d0d4dc6c-a13f-419f-9df2-977e50e6e452" (UID: "d0d4dc6c-a13f-419f-9df2-977e50e6e452"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:01:18.822701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.822673 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5" pod "d0d4dc6c-a13f-419f-9df2-977e50e6e452" (UID: "d0d4dc6c-a13f-419f-9df2-977e50e6e452"). InnerVolumeSpecName "kube-api-access-lnlc5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:18.922650 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.922566 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d0d4dc6c-a13f-419f-9df2-977e50e6e452-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:18.922650 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:18.922599 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnlc5\" (UniqueName: \"kubernetes.io/projected/d0d4dc6c-a13f-419f-9df2-977e50e6e452-kube-api-access-lnlc5\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:19.650614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:19.648742 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" event={"ID":"d0d4dc6c-a13f-419f-9df2-977e50e6e452","Type":"ContainerDied","Data":"ab0b9485e749fd4ab4515f260343d323c90f9f3a176105a0048a0b919be7af6c"} Oct 30 06:01:19.650614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:19.648812 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab0b9485e749fd4ab4515f260343d323c90f9f3a176105a0048a0b919be7af6c" Oct 30 06:01:19.650614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:19.648893 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7" Oct 30 06:01:20.975963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:20.975923 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5"] Oct 30 06:01:20.977158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:20.977129 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d0d4dc6c-a13f-419f-9df2-977e50e6e452" containerName="revoke-keys" Oct 30 06:01:20.977158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:20.977158 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d4dc6c-a13f-419f-9df2-977e50e6e452" containerName="revoke-keys" Oct 30 06:01:20.977495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:20.977478 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d4dc6c-a13f-419f-9df2-977e50e6e452" containerName="revoke-keys" Oct 30 06:01:20.982286 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:20.982231 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.004461 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.004431 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5"] Oct 30 06:01:21.012573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.012513 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d917a1-openshift-default-dockercfg-z6r76\"" Oct 30 06:01:21.089686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.089645 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.089891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.089707 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.089970 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.089881 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.090097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.090064 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.090229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.090188 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5s269\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.090229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.090220 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.090350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.090252 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.090350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.090311 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.090466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.090402 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212312 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212520 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5s269\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212581 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212615 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212681 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212713 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212828 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212877 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.212922 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.213630 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.213892 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.231606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.230992 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.232855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.232824 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.273993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.243763 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.274320 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.249472 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.274448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.267577 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.282609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.282520 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.306125 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.292277 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5s269\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269\") pod \"e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.306125 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.304884 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:21.666851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.666822 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5"] Oct 30 06:01:21.668569 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:01:21.668502 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60308810_612a_4400_b5d3_eeea99079add.slice/crio-8c43b82220599eda78d4327dfbcb21ef5c326dfe4318e07a11ea3b225c316f10 WatchSource:0}: Error finding container 8c43b82220599eda78d4327dfbcb21ef5c326dfe4318e07a11ea3b225c316f10: Status 404 returned error can't find the container with id 8c43b82220599eda78d4327dfbcb21ef5c326dfe4318e07a11ea3b225c316f10 Oct 30 06:01:21.671859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.671822 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:01:21.671943 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.671913 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:01:21.671978 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:21.671958 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:01:22.680879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:22.680845 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" event={"ID":"60308810-612a-4400-b5d3-eeea99079add","Type":"ContainerStarted","Data":"6265d208d55a75f7ccb9bd4abd198bca005775b47d4581696816740c488960a6"} Oct 30 06:01:22.681380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:22.680910 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" event={"ID":"60308810-612a-4400-b5d3-eeea99079add","Type":"ContainerStarted","Data":"8c43b82220599eda78d4327dfbcb21ef5c326dfe4318e07a11ea3b225c316f10"} Oct 30 06:01:22.748261 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:22.748208 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podStartSLOduration=2.74819379 podStartE2EDuration="2.74819379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:22.745495014 +0000 UTC m=+1789.321182068" watchObservedRunningTime="2026-10-30 06:01:22.74819379 +0000 UTC m=+1789.323880846" Oct 30 06:01:23.305388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:23.305351 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:23.312019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:23.311987 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:23.688963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:23.688871 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:23.689863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:23.689841 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:01:25.605965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.605934 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh"] Oct 30 06:01:25.610670 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.610647 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:25.631731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.631706 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-02ceed\"" Oct 30 06:01:25.643867 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.643847 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-02ceed-dockercfg-spcvl\"" Oct 30 06:01:25.648663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.648636 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh"] Oct 30 06:01:25.702364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.702327 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:25.702793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.702762 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj246\" (UniqueName: \"kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246\") pod \"payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:25.829318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.829279 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj246\" (UniqueName: \"kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246\") pod \"payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:25.830246 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.830217 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:25.831383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.831357 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:25.837031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.837008 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d"] Oct 30 06:01:25.853144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.853117 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:25.879363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.879263 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d"] Oct 30 06:01:25.942611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.934000 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs7m5\" (UniqueName: \"kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5\") pod \"payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:25.942611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.934090 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:25.946995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:25.946964 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj246\" (UniqueName: \"kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246\") pod \"payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:26.035711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.035674 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs7m5\" (UniqueName: \"kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5\") pod \"payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:26.035936 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.035749 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:26.036629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.036605 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:26.061122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.061092 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs7m5\" (UniqueName: \"kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5\") pod \"payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:26.225763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.225676 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:26.252385 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.252349 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:26.559516 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:01:26.559424 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d89bf4_20bc_43ef_b5b9_6fb561a77b0a.slice/crio-e7730201baa1473855dd8d4d99e23d696f61d71c45d53278e507acd0fa4629e5 WatchSource:0}: Error finding container e7730201baa1473855dd8d4d99e23d696f61d71c45d53278e507acd0fa4629e5: Status 404 returned error can't find the container with id e7730201baa1473855dd8d4d99e23d696f61d71c45d53278e507acd0fa4629e5 Oct 30 06:01:26.560186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.560157 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh"] Oct 30 06:01:26.607357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.607336 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d"] Oct 30 06:01:26.917522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.917433 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" event={"ID":"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c","Type":"ContainerStarted","Data":"98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a"} Oct 30 06:01:26.917522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.917493 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" event={"ID":"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c","Type":"ContainerStarted","Data":"6112dfbf2e365d9b8e34fd276228785e6afabc92eec32586ecc43e1cc9420bae"} Oct 30 06:01:26.917778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.917564 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:26.919586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.919559 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" event={"ID":"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a","Type":"ContainerStarted","Data":"23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8"} Oct 30 06:01:26.919816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.919787 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" event={"ID":"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a","Type":"ContainerStarted","Data":"e7730201baa1473855dd8d4d99e23d696f61d71c45d53278e507acd0fa4629e5"} Oct 30 06:01:26.919931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.919921 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:26.969016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:26.968967 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" podStartSLOduration=1.968954007 podStartE2EDuration="1.968954007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:26.964436696 +0000 UTC m=+1793.540123752" watchObservedRunningTime="2026-10-30 06:01:26.968954007 +0000 UTC m=+1793.544641104" Oct 30 06:01:27.017307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:27.017256 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" podStartSLOduration=2.017243152 podStartE2EDuration="2.017243152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:01:27.014693649 +0000 UTC m=+1793.590380703" watchObservedRunningTime="2026-10-30 06:01:27.017243152 +0000 UTC m=+1793.592930206" Oct 30 06:01:37.549159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:37.549126 5565 scope.go:122] "RemoveContainer" containerID="e4dc20453fd5a0252c8d1f8cd53c268bb82f459afa4f45182f4e6384aecab40e" Oct 30 06:01:37.563553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:37.563504 5565 scope.go:122] "RemoveContainer" containerID="88b535f5d61ab7770a7ebacc1f4fed906f3747aaa61a5e38756c77e436d1d9f2" Oct 30 06:01:37.576291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:37.576265 5565 scope.go:122] "RemoveContainer" containerID="681008156ac074b27b88dfda9f0bf82fdea639d56021f1a9d9274e6cd5273437" Oct 30 06:01:37.931153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:37.931071 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:01:37.931153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:37.931145 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:01:45.520781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.520580 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd"] Oct 30 06:01:45.530144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.530114 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.542006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.541970 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd"] Oct 30 06:01:45.548392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.548115 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:01:45.548392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.548224 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:01:45.621256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.621229 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.621551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.621400 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bcft\" (UniqueName: \"kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.621551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.621499 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.723208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.723177 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.723356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.723271 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bcft\" (UniqueName: \"kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.723356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.723314 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.723937 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.723913 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.725640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.725619 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.755264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.755238 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bcft\" (UniqueName: \"kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft\") pod \"maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:45.856426 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:45.856351 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:46.189569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:46.184892 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd"] Oct 30 06:01:46.191681 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:01:46.191618 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f5bcc6_7c05_410f_b406_944f62c8b6d3.slice/crio-b90fae727d8a49f4943a62a4aebc69b2654f22ec11ee9a332d6b202e2fcceec2 WatchSource:0}: Error finding container b90fae727d8a49f4943a62a4aebc69b2654f22ec11ee9a332d6b202e2fcceec2: Status 404 returned error can't find the container with id b90fae727d8a49f4943a62a4aebc69b2654f22ec11ee9a332d6b202e2fcceec2 Oct 30 06:01:46.472658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:46.472625 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" event={"ID":"31f5bcc6-7c05-410f-b406-944f62c8b6d3","Type":"ContainerStarted","Data":"b90fae727d8a49f4943a62a4aebc69b2654f22ec11ee9a332d6b202e2fcceec2"} Oct 30 06:01:47.485148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:47.485052 5565 generic.go:356] "Generic (PLEG): container finished" podID="31f5bcc6-7c05-410f-b406-944f62c8b6d3" containerID="5b8887094aecbe38e62297c3b415bffdf128c82c54f77b0affb9c7b44ed6853d" exitCode=0 Oct 30 06:01:47.485575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:47.485134 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" event={"ID":"31f5bcc6-7c05-410f-b406-944f62c8b6d3","Type":"ContainerDied","Data":"5b8887094aecbe38e62297c3b415bffdf128c82c54f77b0affb9c7b44ed6853d"} Oct 30 06:01:47.537213 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:01:47.537172 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:01:48.687577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.687546 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:48.888785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.888744 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft\" (UniqueName: \"kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft\") pod \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " Oct 30 06:01:48.889029 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.888874 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca\") pod \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " Oct 30 06:01:48.889029 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.888904 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca\") pod \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\" (UID: \"31f5bcc6-7c05-410f-b406-944f62c8b6d3\") " Oct 30 06:01:48.889241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.889213 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca" pod "31f5bcc6-7c05-410f-b406-944f62c8b6d3" (UID: "31f5bcc6-7c05-410f-b406-944f62c8b6d3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:01:48.889864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.889840 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:48.891215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.891178 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca" pod "31f5bcc6-7c05-410f-b406-944f62c8b6d3" (UID: "31f5bcc6-7c05-410f-b406-944f62c8b6d3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:01:48.891327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.891219 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft" pod "31f5bcc6-7c05-410f-b406-944f62c8b6d3" (UID: "31f5bcc6-7c05-410f-b406-944f62c8b6d3"). InnerVolumeSpecName "kube-api-access-4bcft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:48.991065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.991018 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bcft\" (UniqueName: \"kubernetes.io/projected/31f5bcc6-7c05-410f-b406-944f62c8b6d3-kube-api-access-4bcft\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:48.991065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:48.991058 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31f5bcc6-7c05-410f-b406-944f62c8b6d3-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:49.621337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:49.621301 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" Oct 30 06:01:49.623017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:49.622979 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd" event={"ID":"31f5bcc6-7c05-410f-b406-944f62c8b6d3","Type":"ContainerDied","Data":"b90fae727d8a49f4943a62a4aebc69b2654f22ec11ee9a332d6b202e2fcceec2"} Oct 30 06:01:49.623212 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:49.623039 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b90fae727d8a49f4943a62a4aebc69b2654f22ec11ee9a332d6b202e2fcceec2" Oct 30 06:01:51.524592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:51.523730 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc"] Oct 30 06:01:51.524592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:51.524011 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" containerID="cri-o://ad76c4a42fd12d36b82e4e48cff476849379a29ecac9e775656fa5561f550910" gracePeriod=30 Oct 30 06:01:55.919707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:55.919667 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825"] Oct 30 06:01:55.964506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:55.964470 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk"] Oct 30 06:01:56.756756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:56.754453 5565 generic.go:356] "Generic (PLEG): container finished" podID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerID="ad76c4a42fd12d36b82e4e48cff476849379a29ecac9e775656fa5561f550910" exitCode=0 Oct 30 06:01:56.756756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:56.754786 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" event={"ID":"c6f93246-910c-4e9d-9a41-386b3b8e00c7","Type":"ContainerDied","Data":"ad76c4a42fd12d36b82e4e48cff476849379a29ecac9e775656fa5561f550910"} Oct 30 06:01:57.016915 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.016884 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 06:01:57.062134 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062103 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062201 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062274 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062302 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062385 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062478 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062660 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:01:57.062769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062743 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:01:57.062832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062778 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.062899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.062876 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:01:57.063155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.063119 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:01:57.063294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.063158 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.063294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.063208 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket\") pod \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\" (UID: \"c6f93246-910c-4e9d-9a41-386b3b8e00c7\") " Oct 30 06:01:57.063777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.063738 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:01:57.064182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.064159 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.064182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.064181 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.064409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.064190 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.064409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.064199 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.064409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.064208 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.065128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.065096 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:01:57.065235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.065144 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "kube-api-access-vgfmw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:57.065687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.065664 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:01:57.065772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.065733 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token" pod "c6f93246-910c-4e9d-9a41-386b3b8e00c7" (UID: "c6f93246-910c-4e9d-9a41-386b3b8e00c7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:01:57.165739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.165665 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.165739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.165694 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.165739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.165704 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6f93246-910c-4e9d-9a41-386b3b8e00c7-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.165739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.165713 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgfmw\" (UniqueName: \"kubernetes.io/projected/c6f93246-910c-4e9d-9a41-386b3b8e00c7-kube-api-access-vgfmw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:01:57.764545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.764502 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" event={"ID":"c6f93246-910c-4e9d-9a41-386b3b8e00c7","Type":"ContainerDied","Data":"058294eecac3eaa77eff20eb8541166a996c28ef077ad61766e7980ee73908b8"} Oct 30 06:01:57.764713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.764594 5565 scope.go:122] "RemoveContainer" containerID="ad76c4a42fd12d36b82e4e48cff476849379a29ecac9e775656fa5561f550910" Oct 30 06:01:57.764713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.764509 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc" Oct 30 06:01:57.848272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.848185 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc"] Oct 30 06:01:57.863231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:57.863201 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-96zxc"] Oct 30 06:01:59.570378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:01:59.570347 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" path="/var/lib/kubelet/pods/c6f93246-910c-4e9d-9a41-386b3b8e00c7/volumes" Oct 30 06:02:00.854553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:00.850075 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:02:00.920692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:00.920603 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerName="payload-processing" containerID="cri-o://747d28c8de081f6658fb96b77181711b54eb5760e7cc32bbac2532a539abb732" gracePeriod=25 Oct 30 06:02:00.971166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:00.971118 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" containerName="payload-pre-processing" containerID="cri-o://130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681" gracePeriod=25 Oct 30 06:02:03.011408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:03.011363 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:02:03.011953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:03.011374 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:02:13.011106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.011068 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:02:13.011664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.011068 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:02:13.256963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.256932 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t"] Oct 30 06:02:13.257843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.257821 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" Oct 30 06:02:13.257843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.257842 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" Oct 30 06:02:13.258037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.257877 5565 cpu_manager.go:414] "containerMap: removing container" podUID="31f5bcc6-7c05-410f-b406-944f62c8b6d3" containerName="revoke-keys" Oct 30 06:02:13.258037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.257885 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f5bcc6-7c05-410f-b406-944f62c8b6d3" containerName="revoke-keys" Oct 30 06:02:13.258141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.258121 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6f93246-910c-4e9d-9a41-386b3b8e00c7" containerName="istio-proxy" Oct 30 06:02:13.258201 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.258142 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f5bcc6-7c05-410f-b406-944f62c8b6d3" containerName="revoke-keys" Oct 30 06:02:13.263815 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.263746 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.286343 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.286315 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t"] Oct 30 06:02:13.296243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.296208 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-765de9\"/\"openshift-service-ca.crt\"" Oct 30 06:02:13.296406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.296258 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-765de9\"/\"kube-root-ca.crt\"" Oct 30 06:02:13.296488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.296395 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-765de9\"/\"test-model-765de9-kserve-self-signed-certs\"" Oct 30 06:02:13.296597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.296489 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-765de9\"/\"default-dockercfg-kmw69\"" Oct 30 06:02:13.359868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.359836 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.360009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.359874 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjsml\" (UniqueName: \"kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.360009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.359998 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.360094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.360066 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.360147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.360126 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.360205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.360192 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462251 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462287 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjsml\" (UniqueName: \"kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462334 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462392 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462417 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462458 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462670 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462723 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.462841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.462822 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.464758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.464728 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.464982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.464962 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.522635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.522477 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjsml\" (UniqueName: \"kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml\") pod \"test-model-765de9-kserve-74c4989c8c-fwk2t\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.604140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.603804 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:13.976942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:13.976906 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t"] Oct 30 06:02:13.977541 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:02:13.977507 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod04b05d01_a811_4f7a_9ab1_012db09e0a93.slice/crio-be9e80b13e1dfaa8bb052c33da8fc2ec4da227684304469edb676cda99371a5f WatchSource:0}: Error finding container be9e80b13e1dfaa8bb052c33da8fc2ec4da227684304469edb676cda99371a5f: Status 404 returned error can't find the container with id be9e80b13e1dfaa8bb052c33da8fc2ec4da227684304469edb676cda99371a5f Oct 30 06:02:14.946627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:14.946585 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" event={"ID":"04b05d01-a811-4f7a-9ab1-012db09e0a93","Type":"ContainerStarted","Data":"4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf"} Oct 30 06:02:14.946627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:14.946640 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" event={"ID":"04b05d01-a811-4f7a-9ab1-012db09e0a93","Type":"ContainerStarted","Data":"be9e80b13e1dfaa8bb052c33da8fc2ec4da227684304469edb676cda99371a5f"} Oct 30 06:02:14.994551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:14.994492 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" podStartSLOduration=1.9944801399999998 podStartE2EDuration="1.99448014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:14.993428818 +0000 UTC m=+1841.569115902" watchObservedRunningTime="2026-10-30 06:02:14.99448014 +0000 UTC m=+1841.570167193" Oct 30 06:02:15.605501 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:15.605454 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:16.683968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:16.683937 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd"] Oct 30 06:02:16.706315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:16.706274 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-b3e625-vdwsd"] Oct 30 06:02:17.535456 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:02:17.535411 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:02:17.571515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:17.571486 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f5bcc6-7c05-410f-b406-944f62c8b6d3" path="/var/lib/kubelet/pods/31f5bcc6-7c05-410f-b406-944f62c8b6d3/volumes" Oct 30 06:02:21.019359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.019324 5565 generic.go:356] "Generic (PLEG): container finished" podID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerID="747d28c8de081f6658fb96b77181711b54eb5760e7cc32bbac2532a539abb732" exitCode=0 Oct 30 06:02:21.019808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.019408 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" event={"ID":"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f","Type":"ContainerDied","Data":"747d28c8de081f6658fb96b77181711b54eb5760e7cc32bbac2532a539abb732"} Oct 30 06:02:21.024811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.024782 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9"] Oct 30 06:02:21.031236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.031214 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.060873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.060826 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.061080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.060893 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tffxv\" (UniqueName: \"kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.061080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.060953 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.061205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.061102 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.061205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.061148 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.061353 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.061210 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.061841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.061813 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-086c60\"/\"test-model-086c60-kserve-self-signed-certs\"" Oct 30 06:02:21.062786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.062555 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-086c60\"/\"default-dockercfg-ts566\"" Oct 30 06:02:21.062786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.063104 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9"] Oct 30 06:02:21.062786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.063640 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-086c60\"/\"openshift-service-ca.crt\"" Oct 30 06:02:21.062786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.064840 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-086c60\"/\"kube-root-ca.crt\"" Oct 30 06:02:21.167796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.167757 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.167982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.167844 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.168453 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.168430 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.168569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.168551 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.168768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.168747 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.168948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.168796 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tffxv\" (UniqueName: \"kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.168948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.168873 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.169715 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.169689 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.169857 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.169749 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.172877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.172851 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.173059 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.172900 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.196620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.196500 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tffxv\" (UniqueName: \"kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv\") pod \"test-model-086c60-kserve-689fd654fc-8pcb9\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.220791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.220765 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 06:02:21.277315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.277260 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume" pod "cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" (UID: "cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:21.277564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.276883 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume\") pod \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " Oct 30 06:02:21.278318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.278292 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6\" (UniqueName: \"kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6\") pod \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\" (UID: \"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f\") " Oct 30 06:02:21.327595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.308242 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6" pod "cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" (UID: "cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f"). InnerVolumeSpecName "kube-api-access-m4mw6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:21.327595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.309896 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:21.327595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.309917 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4mw6\" (UniqueName: \"kubernetes.io/projected/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f-kube-api-access-m4mw6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:21.341769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.341744 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 06:02:21.390039 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.389997 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:21.411438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.411397 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume\") pod \"543fd875-5b9d-481d-8746-20aa656d38f5\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " Oct 30 06:02:21.411803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.411748 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr\") pod \"543fd875-5b9d-481d-8746-20aa656d38f5\" (UID: \"543fd875-5b9d-481d-8746-20aa656d38f5\") " Oct 30 06:02:21.411990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.411963 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume" pod "543fd875-5b9d-481d-8746-20aa656d38f5" (UID: "543fd875-5b9d-481d-8746-20aa656d38f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:21.412969 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.412944 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/543fd875-5b9d-481d-8746-20aa656d38f5-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:21.414804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.414773 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr" pod "543fd875-5b9d-481d-8746-20aa656d38f5" (UID: "543fd875-5b9d-481d-8746-20aa656d38f5"). InnerVolumeSpecName "kube-api-access-44bfr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:21.525268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.516416 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-44bfr\" (UniqueName: \"kubernetes.io/projected/543fd875-5b9d-481d-8746-20aa656d38f5-kube-api-access-44bfr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:21.774120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:21.774086 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9"] Oct 30 06:02:21.775792 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:02:21.775749 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5fbbec0e_d71e_46fb_94bb_90eca336c523.slice/crio-8d4ae2cf95da48f98348da2d1c28372ec3b4459860e3a4b666c090aeda79d70d WatchSource:0}: Error finding container 8d4ae2cf95da48f98348da2d1c28372ec3b4459860e3a4b666c090aeda79d70d: Status 404 returned error can't find the container with id 8d4ae2cf95da48f98348da2d1c28372ec3b4459860e3a4b666c090aeda79d70d Oct 30 06:02:22.029090 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.029018 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" Oct 30 06:02:22.029560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.029012 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825" event={"ID":"cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f","Type":"ContainerDied","Data":"daacfe036c7432bb36720dd795ab6bab861ff89c632e8679ada58b4c4ed75548"} Oct 30 06:02:22.029560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.029148 5565 scope.go:122] "RemoveContainer" containerID="747d28c8de081f6658fb96b77181711b54eb5760e7cc32bbac2532a539abb732" Oct 30 06:02:22.030580 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.030559 5565 generic.go:356] "Generic (PLEG): container finished" podID="543fd875-5b9d-481d-8746-20aa656d38f5" containerID="130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681" exitCode=0 Oct 30 06:02:22.030670 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.030602 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" Oct 30 06:02:22.030735 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.030591 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" event={"ID":"543fd875-5b9d-481d-8746-20aa656d38f5","Type":"ContainerDied","Data":"130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681"} Oct 30 06:02:22.030735 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.030727 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk" event={"ID":"543fd875-5b9d-481d-8746-20aa656d38f5","Type":"ContainerDied","Data":"cd486ad40db46bd540f60f36403cb53b3526e3b82b4a404c3d83012bb75d9323"} Oct 30 06:02:22.032817 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.032785 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" event={"ID":"5fbbec0e-d71e-46fb-94bb-90eca336c523","Type":"ContainerStarted","Data":"df72590602472490e1377072c5f733cb9ac0efefed24d6cc920bdbcb552eb08f"} Oct 30 06:02:22.032944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.032838 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" event={"ID":"5fbbec0e-d71e-46fb-94bb-90eca336c523","Type":"ContainerStarted","Data":"8d4ae2cf95da48f98348da2d1c28372ec3b4459860e3a4b666c090aeda79d70d"} Oct 30 06:02:22.040995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.040952 5565 scope.go:122] "RemoveContainer" containerID="130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681" Oct 30 06:02:22.051255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.051237 5565 scope.go:122] "RemoveContainer" containerID="130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681" Oct 30 06:02:22.051522 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:02:22.051501 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681\": container with ID starting with 130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681 not found: ID does not exist" containerID="130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681" Oct 30 06:02:22.051591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.051567 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681"} err="failed to get container status \"130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681\": rpc error: code = NotFound desc = could not find container \"130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681\": container with ID starting with 130da22c3ad285e8c0b0f541241c0e0c2ef025da1c0084cb5276b490f0ae0681 not found: ID does not exist" Oct 30 06:02:22.088114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.088087 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk"] Oct 30 06:02:22.092111 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.092085 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-b3e625-684dd77cf5-5bttk"] Oct 30 06:02:22.135876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.135265 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825"] Oct 30 06:02:22.144331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.144304 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-b3e625-6b9b44f458-8m825"] Oct 30 06:02:22.184504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.184456 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" podStartSLOduration=2.184442633 podStartE2EDuration="2.184442633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:22.182023383 +0000 UTC m=+1848.757710438" watchObservedRunningTime="2026-10-30 06:02:22.184442633 +0000 UTC m=+1848.760129687" Oct 30 06:02:22.391113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:22.391024 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:23.570045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:23.570013 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" path="/var/lib/kubelet/pods/543fd875-5b9d-481d-8746-20aa656d38f5/volumes" Oct 30 06:02:23.570455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:23.570440 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" path="/var/lib/kubelet/pods/cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f/volumes" Oct 30 06:02:23.605391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:23.605357 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:23.619548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:23.619509 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:24.005334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:24.005301 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz"] Oct 30 06:02:24.005616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:24.005586 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" containerID="cri-o://05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3" gracePeriod=30 Oct 30 06:02:24.066828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:24.066796 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:02:25.759157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.759111 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz"] Oct 30 06:02:25.760566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.760523 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerName="payload-processing" Oct 30 06:02:25.760753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.760737 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerName="payload-processing" Oct 30 06:02:25.760977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.760965 5565 cpu_manager.go:414] "containerMap: removing container" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" containerName="payload-pre-processing" Oct 30 06:02:25.761064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.761052 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" containerName="payload-pre-processing" Oct 30 06:02:25.761733 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.761712 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="543fd875-5b9d-481d-8746-20aa656d38f5" containerName="payload-pre-processing" Oct 30 06:02:25.761890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.761876 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd7fabd8-1055-4ab4-8ab5-c37a54eb2b6f" containerName="payload-processing" Oct 30 06:02:25.768731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.768712 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.785195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.785171 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz"] Oct 30 06:02:25.794493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.794469 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:02:25.794641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.794621 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:02:25.815987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.815956 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.816195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.816162 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9fkf\" (UniqueName: \"kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.816334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.816312 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.917641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.917602 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.917900 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.917872 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.918081 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.918064 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9fkf\" (UniqueName: \"kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.918314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.918288 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.920378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.920335 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:25.944335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:25.944307 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9fkf\" (UniqueName: \"kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf\") pod \"maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:26.086434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:26.086362 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:26.420914 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:26.420884 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz"] Oct 30 06:02:26.421618 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:02:26.421592 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1968c155_8bcd_459b_b5ca_241e60f7905f.slice/crio-7769c88c3942607cacc0fe74c38384d7510b88f5d10a969761984e7266b943bc WatchSource:0}: Error finding container 7769c88c3942607cacc0fe74c38384d7510b88f5d10a969761984e7266b943bc: Status 404 returned error can't find the container with id 7769c88c3942607cacc0fe74c38384d7510b88f5d10a969761984e7266b943bc Oct 30 06:02:27.044734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:27.044699 5565 patch_prober.go:29] interesting pod/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:02:27.045108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:27.044752 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:02:27.078716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:27.078684 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" event={"ID":"1968c155-8bcd-459b-b5ca-241e60f7905f","Type":"ContainerStarted","Data":"3ef486838c153efacc501af0d8dd80bd6672a900711be748e7600a884d02251a"} Oct 30 06:02:27.078884 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:27.078744 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" event={"ID":"1968c155-8bcd-459b-b5ca-241e60f7905f","Type":"ContainerStarted","Data":"7769c88c3942607cacc0fe74c38384d7510b88f5d10a969761984e7266b943bc"} Oct 30 06:02:27.124382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:27.124327 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" podStartSLOduration=2.124314305 podStartE2EDuration="2.124314305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:27.122889556 +0000 UTC m=+1853.698576611" watchObservedRunningTime="2026-10-30 06:02:27.124314305 +0000 UTC m=+1853.700001356" Oct 30 06:02:28.088257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:28.088225 5565 generic.go:356] "Generic (PLEG): container finished" podID="1968c155-8bcd-459b-b5ca-241e60f7905f" containerID="3ef486838c153efacc501af0d8dd80bd6672a900711be748e7600a884d02251a" exitCode=0 Oct 30 06:02:28.088726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:28.088320 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" event={"ID":"1968c155-8bcd-459b-b5ca-241e60f7905f","Type":"ContainerDied","Data":"3ef486838c153efacc501af0d8dd80bd6672a900711be748e7600a884d02251a"} Oct 30 06:02:29.300952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.300928 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:29.355466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.355429 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 06:02:29.369950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.369919 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.369987 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca\") pod \"1968c155-8bcd-459b-b5ca-241e60f7905f\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " Oct 30 06:02:29.370101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370020 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370106 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca\") pod \"1968c155-8bcd-459b-b5ca-241e60f7905f\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " Oct 30 06:02:29.370236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370157 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370179 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.370236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370203 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370449 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370257 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf\" (UniqueName: \"kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf\") pod \"1968c155-8bcd-459b-b5ca-241e60f7905f\" (UID: \"1968c155-8bcd-459b-b5ca-241e60f7905f\") " Oct 30 06:02:29.370449 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370297 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370449 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370331 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370449 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370425 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370463 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca" pod "1968c155-8bcd-459b-b5ca-241e60f7905f" (UID: "1968c155-8bcd-459b-b5ca-241e60f7905f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:29.370704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370572 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370659 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy\") pod \"1de9a4c3-0b40-40f1-8f06-518a5382c642\" (UID: \"1de9a4c3-0b40-40f1-8f06-518a5382c642\") " Oct 30 06:02:29.370917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.370896 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:29.371506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.371480 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.371643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.371510 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1de9a4c3-0b40-40f1-8f06-518a5382c642-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.371643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.371516 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.371643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.371538 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.371822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.371759 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.372061 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.372041 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.373316 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.373289 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "kube-api-access-dctkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:29.374000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.373968 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:29.376886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.376861 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:02:29.376886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.376877 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca" pod "1968c155-8bcd-459b-b5ca-241e60f7905f" (UID: "1968c155-8bcd-459b-b5ca-241e60f7905f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:02:29.377082 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.376888 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token" pod "1de9a4c3-0b40-40f1-8f06-518a5382c642" (UID: "1de9a4c3-0b40-40f1-8f06-518a5382c642"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:29.378070 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.378052 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf" pod "1968c155-8bcd-459b-b5ca-241e60f7905f" (UID: "1968c155-8bcd-459b-b5ca-241e60f7905f"). InnerVolumeSpecName "kube-api-access-j9fkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:29.473025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.472988 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1968c155-8bcd-459b-b5ca-241e60f7905f-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473027 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473037 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473049 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9fkf\" (UniqueName: \"kubernetes.io/projected/1968c155-8bcd-459b-b5ca-241e60f7905f-kube-api-access-j9fkf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473059 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473067 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473077 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dctkt\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-kube-api-access-dctkt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473093 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:29.473281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:29.473105 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1de9a4c3-0b40-40f1-8f06-518a5382c642-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:30.105796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.105768 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" Oct 30 06:02:30.105946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.105759 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz" event={"ID":"1968c155-8bcd-459b-b5ca-241e60f7905f","Type":"ContainerDied","Data":"7769c88c3942607cacc0fe74c38384d7510b88f5d10a969761984e7266b943bc"} Oct 30 06:02:30.105946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.105905 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7769c88c3942607cacc0fe74c38384d7510b88f5d10a969761984e7266b943bc" Oct 30 06:02:30.107821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.107792 5565 generic.go:356] "Generic (PLEG): container finished" podID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerID="05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3" exitCode=0 Oct 30 06:02:30.107975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.107850 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" event={"ID":"1de9a4c3-0b40-40f1-8f06-518a5382c642","Type":"ContainerDied","Data":"05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3"} Oct 30 06:02:30.107975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.107856 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" Oct 30 06:02:30.107975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.107888 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz" event={"ID":"1de9a4c3-0b40-40f1-8f06-518a5382c642","Type":"ContainerDied","Data":"e0b2fae70e1917a07ee7b5900a19becb7b41bf54ef0ce971e48956f613732989"} Oct 30 06:02:30.107975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.107917 5565 scope.go:122] "RemoveContainer" containerID="05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3" Oct 30 06:02:30.119388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.119371 5565 scope.go:122] "RemoveContainer" containerID="212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba" Oct 30 06:02:30.129208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.129193 5565 scope.go:122] "RemoveContainer" containerID="05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3" Oct 30 06:02:30.129497 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:02:30.129473 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3\": container with ID starting with 05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3 not found: ID does not exist" containerID="05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3" Oct 30 06:02:30.129610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.129499 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3"} err="failed to get container status \"05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3\": rpc error: code = NotFound desc = could not find container \"05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3\": container with ID starting with 05981111e27654d54638e7df826077029611c699523947722caacc7f8b6ee2b3 not found: ID does not exist" Oct 30 06:02:30.129610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.129523 5565 scope.go:122] "RemoveContainer" containerID="212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba" Oct 30 06:02:30.129829 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:02:30.129810 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba\": container with ID starting with 212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba not found: ID does not exist" containerID="212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba" Oct 30 06:02:30.129869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.129834 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba"} err="failed to get container status \"212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba\": rpc error: code = NotFound desc = could not find container \"212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba\": container with ID starting with 212e12d6c7893f0b30ae0bf8aebff0ef4f2e1d154a8e08d29d445374d0d686ba not found: ID does not exist" Oct 30 06:02:30.185336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.185306 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz"] Oct 30 06:02:30.189935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:30.189910 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-b3e625-openshift-default-6bf65b88b9-ptnfz"] Oct 30 06:02:31.391248 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:31.391213 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:31.405445 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:31.405416 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:31.569909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:31.569882 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" path="/var/lib/kubelet/pods/1de9a4c3-0b40-40f1-8f06-518a5382c642/volumes" Oct 30 06:02:32.138892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:32.138856 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:02:35.399356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:35.399318 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7"] Oct 30 06:02:35.399929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:35.399511 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" containerID="cri-o://0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86" gracePeriod=30 Oct 30 06:02:36.134825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.134788 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt"] Oct 30 06:02:36.135635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135613 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" Oct 30 06:02:36.135635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135634 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" Oct 30 06:02:36.135825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135688 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" Oct 30 06:02:36.135825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135696 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" Oct 30 06:02:36.135825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135728 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1968c155-8bcd-459b-b5ca-241e60f7905f" containerName="revoke-keys" Oct 30 06:02:36.135825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135736 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1968c155-8bcd-459b-b5ca-241e60f7905f" containerName="revoke-keys" Oct 30 06:02:36.136048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135951 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" Oct 30 06:02:36.136048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.135982 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1968c155-8bcd-459b-b5ca-241e60f7905f" containerName="revoke-keys" Oct 30 06:02:36.141510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.141487 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.164835 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.164792 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt"] Oct 30 06:02:36.169949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.169926 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8de592\"/\"kube-root-ca.crt\"" Oct 30 06:02:36.170144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.170057 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8de592\"/\"test-model-8de592-kserve-self-signed-certs\"" Oct 30 06:02:36.182736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.182713 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-8de592\"/\"default-dockercfg-kxz87\"" Oct 30 06:02:36.182736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.182713 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-8de592\"/\"openshift-service-ca.crt\"" Oct 30 06:02:36.245820 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.245780 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.246004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.245908 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p95q8\" (UniqueName: \"kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.246004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.245965 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.246134 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.246006 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.246134 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.246074 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.246254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.246207 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.348821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.348786 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p95q8\" (UniqueName: \"kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.348987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.348860 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.348987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.348900 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.348987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.348963 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.349159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.349086 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.349284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.349269 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.349427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.349394 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.349603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.349563 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.349689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.349636 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.351846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.351823 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.352076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.352057 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.375696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.375673 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p95q8\" (UniqueName: \"kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8\") pod \"test-model-8de592-kserve-d78bd6ccc-j8fjt\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:36.481603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:36.479870 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:37.149857 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:37.149809 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt"] Oct 30 06:02:37.286366 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:37.285505 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" event={"ID":"e30c4190-1c92-415a-9c32-1a48773963d4","Type":"ContainerStarted","Data":"583fae335c9f2bc7fc1c887996250f57e3389acb42408d12c16d993872b8500b"} Oct 30 06:02:37.623876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:37.621077 5565 scope.go:122] "RemoveContainer" containerID="15513c429954fc47b16974d9db4c4043ac29657aa22c9ad2bbfe3f4651d8433b" Oct 30 06:02:38.295576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:38.295538 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" event={"ID":"e30c4190-1c92-415a-9c32-1a48773963d4","Type":"ContainerStarted","Data":"d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e"} Oct 30 06:02:38.352264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:38.352205 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" podStartSLOduration=2.352185792 podStartE2EDuration="2.352185792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:38.348861581 +0000 UTC m=+1864.924548636" watchObservedRunningTime="2026-10-30 06:02:38.352185792 +0000 UTC m=+1864.927872848" Oct 30 06:02:38.482772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:38.481437 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:39.634426 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.634389 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f"] Oct 30 06:02:39.635408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.635387 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1de9a4c3-0b40-40f1-8f06-518a5382c642" containerName="istio-proxy" Oct 30 06:02:39.640753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.640734 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.663888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.663860 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f"] Oct 30 06:02:39.671329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.671306 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-7dfcb7-serving-cert\"" Oct 30 06:02:39.734902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.734869 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.735077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.734911 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.735132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.735095 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfpsx\" (UniqueName: \"kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.735317 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.735298 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.837086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.837052 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfpsx\" (UniqueName: \"kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.837254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.837237 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.837413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.837394 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.837456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.837423 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.838075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.838052 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.839684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.839660 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.839803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.839758 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.884866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.882377 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfpsx\" (UniqueName: \"kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx\") pod \"maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:39.921559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.914731 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:02:39.921559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.914825 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:02:39.956601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:39.956542 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:40.279014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.278984 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f"] Oct 30 06:02:40.279157 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:02:40.278984 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd96d8e_a1fc_4b3d_9192_0994693701c7.slice/crio-e69a12cc040c0dbb4154cbe2e4b7dbb4ba8490668afe603ddfe5672bd8fbeac6 WatchSource:0}: Error finding container e69a12cc040c0dbb4154cbe2e4b7dbb4ba8490668afe603ddfe5672bd8fbeac6: Status 404 returned error can't find the container with id e69a12cc040c0dbb4154cbe2e4b7dbb4ba8490668afe603ddfe5672bd8fbeac6 Oct 30 06:02:40.313593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.313560 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" event={"ID":"5cd96d8e-a1fc-4b3d-9192-0994693701c7","Type":"ContainerStarted","Data":"e69a12cc040c0dbb4154cbe2e4b7dbb4ba8490668afe603ddfe5672bd8fbeac6"} Oct 30 06:02:40.694702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.694672 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 06:02:40.758043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758011 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758154 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758273 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758209 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758323 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:40.758413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758354 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758451 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:40.758608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758590 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758719 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758835 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758820 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.758962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758938 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.759021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.758988 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:40.759080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.759027 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert\") pod \"588bcdf5-b140-4542-9d75-30ca04152ab8\" (UID: \"588bcdf5-b140-4542-9d75-30ca04152ab8\") " Oct 30 06:02:40.759729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.759501 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:40.759906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.759868 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:02:40.760318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.760287 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.760318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.760317 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.760508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.760330 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/588bcdf5-b140-4542-9d75-30ca04152ab8-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.760508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.760351 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.760508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.760363 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.761084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.761056 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:02:40.761258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.761238 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "kube-api-access-n6mvw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:40.761563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.761508 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:02:40.763105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.763077 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token" pod "588bcdf5-b140-4542-9d75-30ca04152ab8" (UID: "588bcdf5-b140-4542-9d75-30ca04152ab8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:02:40.864133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.863289 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.864133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.863340 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.864133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.863355 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6mvw\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-kube-api-access-n6mvw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:40.864133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:40.863426 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/588bcdf5-b140-4542-9d75-30ca04152ab8-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:02:41.323894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.323856 5565 generic.go:356] "Generic (PLEG): container finished" podID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerID="0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86" exitCode=0 Oct 30 06:02:41.324083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.323939 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" Oct 30 06:02:41.324083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.323940 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" event={"ID":"588bcdf5-b140-4542-9d75-30ca04152ab8","Type":"ContainerDied","Data":"0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86"} Oct 30 06:02:41.324194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.324095 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7" event={"ID":"588bcdf5-b140-4542-9d75-30ca04152ab8","Type":"ContainerDied","Data":"0bb5db0d9298e80e9e8a163235092cf370d4d9f951c2ecd779580f53f088b039"} Oct 30 06:02:41.324194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.324126 5565 scope.go:122] "RemoveContainer" containerID="0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86" Oct 30 06:02:41.326221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.326194 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" event={"ID":"5cd96d8e-a1fc-4b3d-9192-0994693701c7","Type":"ContainerStarted","Data":"0a7bd610e6059a1f92e6ae81aec7dae15290997fd00143df0d6c36656bdc8028"} Oct 30 06:02:41.326343 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.326295 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:41.336398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.336379 5565 scope.go:122] "RemoveContainer" containerID="0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86" Oct 30 06:02:41.336776 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:02:41.336749 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86\": container with ID starting with 0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86 not found: ID does not exist" containerID="0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86" Oct 30 06:02:41.336868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.336788 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86"} err="failed to get container status \"0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86\": rpc error: code = NotFound desc = could not find container \"0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86\": container with ID starting with 0ce6f7c0b44819c9d6280984b6530c3aeff16b69be000f8a478ee808bb7e1c86 not found: ID does not exist" Oct 30 06:02:41.394387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.394343 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" podStartSLOduration=2.118390441 podStartE2EDuration="2.394330009s" totalImagesPullingTime="275.939568ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:02:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:02:41.391891928 +0000 UTC m=+1867.967578984" watchObservedRunningTime="2026-10-30 06:02:41.394330009 +0000 UTC m=+1867.970017063" Oct 30 06:02:41.435897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.435867 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7"] Oct 30 06:02:41.439822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.439799 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-nzvj7"] Oct 30 06:02:41.581998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:41.581921 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" path="/var/lib/kubelet/pods/588bcdf5-b140-4542-9d75-30ca04152ab8/volumes" Oct 30 06:02:46.494737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:46.494704 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:46.511285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:46.511256 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:47.341284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:47.341252 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:02:47.391232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:02:47.391199 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:02:47.537373 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:02:47.537329 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:03:01.578414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:01.578379 5565 generic.go:356] "Generic (PLEG): container finished" podID="60308810-612a-4400-b5d3-eeea99079add" containerID="6265d208d55a75f7ccb9bd4abd198bca005775b47d4581696816740c488960a6" exitCode=0 Oct 30 06:03:01.578953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:01.578450 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" event={"ID":"60308810-612a-4400-b5d3-eeea99079add","Type":"ContainerDied","Data":"6265d208d55a75f7ccb9bd4abd198bca005775b47d4581696816740c488960a6"} Oct 30 06:03:01.578953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:01.578920 5565 scope.go:122] "RemoveContainer" containerID="6265d208d55a75f7ccb9bd4abd198bca005775b47d4581696816740c488960a6" Oct 30 06:03:01.580758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:01.580729 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:01.580851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:01.580796 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:01.580851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:01.580831 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:02.598351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:02.598317 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" event={"ID":"60308810-612a-4400-b5d3-eeea99079add","Type":"ContainerStarted","Data":"7f6261a4f063267eb28d746b394a3b2ef3a91d3ef4dd834378d9089d5fcb2761"} Oct 30 06:03:03.306034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:03.306003 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:03:03.306246 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:03.306221 5565 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:03:03.306340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:03.306288 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:03:04.305546 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:04.305502 5565 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:03:04.305968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:04.305588 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:03:05.306468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:05.306433 5565 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:03:05.306902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:05.306487 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:03:06.306508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:06.306474 5565 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:03:06.306909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:06.306540 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:03:07.305643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:07.305612 5565 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body= Oct 30 06:03:07.305899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:07.305662 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" Oct 30 06:03:08.310891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:08.310861 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:03:08.311409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:08.311389 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:03:08.312027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:08.312006 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:03:09.792583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.792504 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp"] Oct 30 06:03:09.793501 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.793476 5565 cpu_manager.go:414] "containerMap: removing container" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" Oct 30 06:03:09.793501 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.793501 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" Oct 30 06:03:09.793805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.793782 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="588bcdf5-b140-4542-9d75-30ca04152ab8" containerName="istio-proxy" Oct 30 06:03:09.798122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.798104 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819022 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.818997 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819150 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819056 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819161 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819250 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819227 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819311 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819250 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819335 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bp76\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819540 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819573 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.819722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.819697 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.825650 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.825626 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp"] Oct 30 06:03:09.826295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.826278 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-765de9-openshift-default-dockercfg-72jrs\"" Oct 30 06:03:09.920900 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.920867 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.920911 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.920962 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921048 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921089 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921118 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921181 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921203 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921250 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bp76\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921290 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921413 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921494 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921684 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.921890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.921873 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.923485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.923459 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.923734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.923716 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.950307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.950277 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bp76\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:09.951339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:09.951320 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token\") pod \"e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:10.113490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.113395 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:10.442989 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:03:10.442960 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48b14bc0_c5dd_4f8c_b29a_62da80c63a93.slice/crio-659237ea16fc88ddb56f71595468b429cb4a015cbfe4e6df6ee30a4a6f7a3497 WatchSource:0}: Error finding container 659237ea16fc88ddb56f71595468b429cb4a015cbfe4e6df6ee30a4a6f7a3497: Status 404 returned error can't find the container with id 659237ea16fc88ddb56f71595468b429cb4a015cbfe4e6df6ee30a4a6f7a3497 Oct 30 06:03:10.443417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.443392 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp"] Oct 30 06:03:10.445800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.445766 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:10.445922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.445844 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:10.445922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.445916 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:10.666354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.666314 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" event={"ID":"48b14bc0-c5dd-4f8c-b29a-62da80c63a93","Type":"ContainerStarted","Data":"e943d8064d3934fc0cc779d5993604cbbebd3f233772dbe09b13db348c2c860d"} Oct 30 06:03:10.666555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.666383 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" event={"ID":"48b14bc0-c5dd-4f8c-b29a-62da80c63a93","Type":"ContainerStarted","Data":"659237ea16fc88ddb56f71595468b429cb4a015cbfe4e6df6ee30a4a6f7a3497"} Oct 30 06:03:10.726638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:10.726519 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podStartSLOduration=1.726500404 podStartE2EDuration="1.726500404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:10.723013707 +0000 UTC m=+1897.298700773" watchObservedRunningTime="2026-10-30 06:03:10.726500404 +0000 UTC m=+1897.302187461" Oct 30 06:03:11.114473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:11.114439 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:11.116205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:11.116175 5565 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:03:11.116369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:11.116231 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 06:03:12.114031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:12.113995 5565 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:03:12.114221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:12.114051 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 06:03:13.113801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:13.113763 5565 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:03:13.114296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:13.113821 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 06:03:14.113846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:14.113812 5565 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:03:14.114288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:14.113864 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 06:03:15.114041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:15.113998 5565 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:03:15.115913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:15.115869 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 06:03:16.114575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:16.114509 5565 patch_prober.go:29] interesting pod/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" start-of-body= Oct 30 06:03:16.114954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:16.114592 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.205:15021/healthz/ready\": dial tcp 10.134.0.205:15021: connect: connection refused" Oct 30 06:03:17.410998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:17.410963 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:17.411719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:17.411694 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:17.412939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:17.412918 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:03:17.536946 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:03:17.536908 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:03:32.612382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:32.612347 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz"] Oct 30 06:03:32.626201 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:32.626174 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-39147c-7tvzz"] Oct 30 06:03:33.570604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:33.570570 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1968c155-8bcd-459b-b5ca-241e60f7905f" path="/var/lib/kubelet/pods/1968c155-8bcd-459b-b5ca-241e60f7905f/volumes" Oct 30 06:03:35.813494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:35.813451 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:03:37.761548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:37.758806 5565 scope.go:122] "RemoveContainer" containerID="5ba8428aec1e1da4c25e6fcbea489550bf90d361a938fbf006d280dfeda3bd56" Oct 30 06:03:37.795106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:37.795081 5565 scope.go:122] "RemoveContainer" containerID="9050be9726838af12724cd5f8e6a85dc46393b73005ba19d45a0f0f2cbe9b8bf" Oct 30 06:03:37.810436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:37.810319 5565 scope.go:122] "RemoveContainer" containerID="5878c4813d410bf2c349aa75a6b507d8df26f0489c930282db8224972422701e" Oct 30 06:03:43.530018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.529982 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66"] Oct 30 06:03:43.538019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.537987 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.573796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.573755 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66"] Oct 30 06:03:43.617687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.617656 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-080391-openshift-default-dockercfg-w5wmr\"" Oct 30 06:03:43.762589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.755789 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.762589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.755870 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.762589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.755949 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n8c7\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.762589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.756000 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.805685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.756052 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.805685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.772409 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.805685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.772518 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.805685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.772732 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.805685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.772782 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.876641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.876404 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.877591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.877323 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.878329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.877050 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.878884 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.878265 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.879039 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.878516 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n8c7\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.879220 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.879200 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.879811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.879792 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.880352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.880322 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.880921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.880890 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.881236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.881217 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.881746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.881721 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.886516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.881645 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.886712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.880194 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.886850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.885270 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.886975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.886155 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.894173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.894147 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.919619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.919550 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n8c7\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:43.924077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:43.924045 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token\") pod \"e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:44.153184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:44.153100 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:44.580710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:44.580685 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66"] Oct 30 06:03:44.581909 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:03:44.581887 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5b7b1b7_e95c_46b9_b61a_86fb98a3c6ea.slice/crio-30aee0f9256f6969a319e0c2beefa4bfa386628b4970d68a7e6a1ba6952af1bd WatchSource:0}: Error finding container 30aee0f9256f6969a319e0c2beefa4bfa386628b4970d68a7e6a1ba6952af1bd: Status 404 returned error can't find the container with id 30aee0f9256f6969a319e0c2beefa4bfa386628b4970d68a7e6a1ba6952af1bd Oct 30 06:03:44.584782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:44.584748 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:44.584859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:44.584818 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:44.584859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:44.584847 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:03:45.457071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:45.457034 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" event={"ID":"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea","Type":"ContainerStarted","Data":"31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472"} Oct 30 06:03:45.457262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:45.457091 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" event={"ID":"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea","Type":"ContainerStarted","Data":"30aee0f9256f6969a319e0c2beefa4bfa386628b4970d68a7e6a1ba6952af1bd"} Oct 30 06:03:45.525797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:45.525746 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podStartSLOduration=2.525732268 podStartE2EDuration="2.525732268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:45.522196977 +0000 UTC m=+1932.097884052" watchObservedRunningTime="2026-10-30 06:03:45.525732268 +0000 UTC m=+1932.101419331" Oct 30 06:03:46.153769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:46.153739 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:46.160364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:46.160327 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:46.464771 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:46.464666 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:46.465662 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:46.465641 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:03:47.533590 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:03:47.533501 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:03:50.230511 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.230463 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz"] Oct 30 06:03:50.236782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.236752 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.271382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.271351 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz"] Oct 30 06:03:50.275100 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.275075 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-080391\"" Oct 30 06:03:50.275310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.275291 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-080391-dockercfg-mp8bd\"" Oct 30 06:03:50.475170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.475134 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmnnx\" (UniqueName: \"kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx\") pod \"payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.494606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.476176 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.619561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.605067 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmnnx\" (UniqueName: \"kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx\") pod \"payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.619945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.619904 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.663951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.627620 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.692615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.689024 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmnnx\" (UniqueName: \"kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx\") pod \"payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:50.856542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:50.856508 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:51.194708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:51.194662 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz"] Oct 30 06:03:51.673500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:51.673459 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" event={"ID":"bd12a404-1cde-4e6e-96e0-38035239d979","Type":"ContainerStarted","Data":"36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60"} Oct 30 06:03:51.673974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:51.673535 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" event={"ID":"bd12a404-1cde-4e6e-96e0-38035239d979","Type":"ContainerStarted","Data":"7e8983e5256e4f194a80688eacecde40b6f01d4e844a7148a74ed184aa5c19ab"} Oct 30 06:03:51.673974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:51.673568 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:03:51.724860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:03:51.724793 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" podStartSLOduration=1.724779193 podStartE2EDuration="1.724779193s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:03:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:03:51.723071318 +0000 UTC m=+1938.298758373" watchObservedRunningTime="2026-10-30 06:03:51.724779193 +0000 UTC m=+1938.300466246" Oct 30 06:04:02.684362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:02.684320 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:04:08.391325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.391283 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr"] Oct 30 06:04:08.401915 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.401877 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.417898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.417871 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr"] Oct 30 06:04:08.451853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.451809 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8de592-openshift-default-dockercfg-kzz2m\"" Oct 30 06:04:08.484117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484082 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484165 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484313 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484465 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484510 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484652 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfr75\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484733 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484714 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484779 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.484860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.484842 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.587117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587080 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.587336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587147 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.587336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587201 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587659 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587720 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587777 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.587930 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588045 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588075 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588262 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588418 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfr75\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588486 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588625 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.589630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.588650 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.590929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.590639 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.591476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.591454 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.619124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.619095 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.619324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.619303 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfr75\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75\") pod \"e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:08.723870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:08.723796 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:09.100875 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:04:09.100836 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc821543_7000_4f3a_bf81_e8d0e938a8cf.slice/crio-6588411dbf39a8feea6a5960c9fcbb6b9fb2303eb647cfd0732b3d844b35d1c5 WatchSource:0}: Error finding container 6588411dbf39a8feea6a5960c9fcbb6b9fb2303eb647cfd0732b3d844b35d1c5: Status 404 returned error can't find the container with id 6588411dbf39a8feea6a5960c9fcbb6b9fb2303eb647cfd0732b3d844b35d1c5 Oct 30 06:04:09.101221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.101190 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr"] Oct 30 06:04:09.103751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.103722 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:04:09.103833 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.103786 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:04:09.103833 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.103821 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:04:09.887346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.887314 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" event={"ID":"bc821543-7000-4f3a-bf81-e8d0e938a8cf","Type":"ContainerStarted","Data":"5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4"} Oct 30 06:04:09.887847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.887370 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" event={"ID":"bc821543-7000-4f3a-bf81-e8d0e938a8cf","Type":"ContainerStarted","Data":"6588411dbf39a8feea6a5960c9fcbb6b9fb2303eb647cfd0732b3d844b35d1c5"} Oct 30 06:04:09.972172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:09.972123 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podStartSLOduration=1.972106988 podStartE2EDuration="1.972106988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:04:09.968799205 +0000 UTC m=+1956.544486260" watchObservedRunningTime="2026-10-30 06:04:09.972106988 +0000 UTC m=+1956.547794045" Oct 30 06:04:10.724514 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:10.724475 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:10.726159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:10.726127 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 06:04:10.726294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:10.726189 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 06:04:11.724722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:11.724679 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 06:04:11.725161 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:11.724761 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 06:04:12.724477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:12.724447 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 06:04:12.724764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:12.724499 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 06:04:13.724929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:13.724892 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 06:04:13.725433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:13.724964 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 06:04:14.724158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:14.724124 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 06:04:14.724343 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:14.724183 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 06:04:15.724852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:15.724812 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" start-of-body= Oct 30 06:04:15.725334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:15.724874 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.208:15021/healthz/ready\": dial tcp 10.134.0.208:15021: connect: connection refused" Oct 30 06:04:16.729374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:16.729337 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:16.729787 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:16.729751 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:16.730459 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:16.730443 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:04:17.534231 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:04:17.534190 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:04:33.037112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:33.037076 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp"] Oct 30 06:04:33.050335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:33.050306 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0268343a-8t6sp"] Oct 30 06:04:33.571375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:33.571344 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20edd61a-214b-42bd-a4cb-df979476fa73" path="/var/lib/kubelet/pods/20edd61a-214b-42bd-a4cb-df979476fa73/volumes" Oct 30 06:04:37.869443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:37.869410 5565 scope.go:122] "RemoveContainer" containerID="f1e6521f2ca93ecbd4f667867a49658a6779e65a998bee0179073618b81af730" Oct 30 06:04:37.880572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:37.880540 5565 scope.go:122] "RemoveContainer" containerID="578fa8ef80fd9b5663eb3090b011d61bd9a7e8c4624eae96be4b6285fc6f31d9" Oct 30 06:04:37.890210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:37.890185 5565 scope.go:122] "RemoveContainer" containerID="4a01a3e3281f88861e4911a07c523421aafbe046b6efa4990401749741bcf10a" Oct 30 06:04:37.900026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:04:37.900003 5565 scope.go:122] "RemoveContainer" containerID="0a6015da1b44922a8984faf5a2a707f201217d5d81ed2aac1ce26228cf6e60b6" Oct 30 06:04:47.538163 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:04:47.538072 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:05:00.156893 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:00.156851 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:05:01.525638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:01.523842 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8"] Oct 30 06:05:01.525638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:01.524097 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" containerID="cri-o://9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f" gracePeriod=30 Oct 30 06:05:03.119247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:03.119200 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z"] Oct 30 06:05:03.120019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:03.119984 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" containerID="cri-o://4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0" gracePeriod=30 Oct 30 06:05:05.526427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:05.526393 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m"] Oct 30 06:05:05.526918 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:05.526687 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" containerID="cri-o://86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5" gracePeriod=30 Oct 30 06:05:05.802627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:05.802536 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2"] Oct 30 06:05:05.802816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:05.802776 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" containerID="cri-o://e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1" gracePeriod=30 Oct 30 06:05:06.928342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:06.928299 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 06:05:07.026245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026211 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.026436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026286 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.026436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026338 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.026592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026470 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.026592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026510 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.026876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026805 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:07.026876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026812 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:07.027177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026913 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.027177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026968 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:07.027177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.026980 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.027177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027045 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.027177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027087 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy\") pod \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\" (UID: \"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e\") " Oct 30 06:05:07.027466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027183 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:07.027466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027387 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:07.027901 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027876 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.028013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027905 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.028013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027921 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.028013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027938 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.028013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.027950 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.029864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.029833 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:07.030092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.030055 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "kube-api-access-tlkfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:07.030232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.030206 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:07.030333 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.030196 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token" pod "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" (UID: "76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:07.129926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.129891 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.129926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.129923 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlkfp\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-kube-api-access-tlkfp\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.129926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.129934 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.130180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.129945 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:07.172840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.172759 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf"] Oct 30 06:05:07.173075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.173024 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" containerID="cri-o://b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692" gracePeriod=30 Oct 30 06:05:07.525425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.525383 5565 generic.go:356] "Generic (PLEG): container finished" podID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerID="9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f" exitCode=0 Oct 30 06:05:07.525623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.525494 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" Oct 30 06:05:07.525623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.525500 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" event={"ID":"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e","Type":"ContainerDied","Data":"9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f"} Oct 30 06:05:07.525623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.525595 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8" event={"ID":"76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e","Type":"ContainerDied","Data":"a9161c1e7e1aa08dd751e9d52112cbde6ca936b95a001c4d3c50f62afb80ebbf"} Oct 30 06:05:07.525762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.525625 5565 scope.go:122] "RemoveContainer" containerID="9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f" Oct 30 06:05:07.538434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.538414 5565 scope.go:122] "RemoveContainer" containerID="c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a" Oct 30 06:05:07.549027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.549010 5565 scope.go:122] "RemoveContainer" containerID="9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f" Oct 30 06:05:07.549348 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:07.549323 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f\": container with ID starting with 9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f not found: ID does not exist" containerID="9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f" Oct 30 06:05:07.549589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.549366 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f"} err="failed to get container status \"9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f\": rpc error: code = NotFound desc = could not find container \"9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f\": container with ID starting with 9245b2b40aabd29a078c4375927130175c53cbcd3cbb8869a6977b0af171029f not found: ID does not exist" Oct 30 06:05:07.549589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.549395 5565 scope.go:122] "RemoveContainer" containerID="c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a" Oct 30 06:05:07.549726 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:07.549684 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a\": container with ID starting with c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a not found: ID does not exist" containerID="c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a" Oct 30 06:05:07.549726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.549707 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a"} err="failed to get container status \"c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a\": rpc error: code = NotFound desc = could not find container \"c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a\": container with ID starting with c9e88a892da8a0e8ccb9186df45cd4ff0fdb571b52e5a67e6fd16634c515359a not found: ID does not exist" Oct 30 06:05:07.597624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.597590 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8"] Oct 30 06:05:07.601567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:07.601523 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-37bbcc-openshift-default-8c9754577-jvxj8"] Oct 30 06:05:08.480455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.480430 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 06:05:08.542137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.542098 5565 generic.go:356] "Generic (PLEG): container finished" podID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerID="4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0" exitCode=0 Oct 30 06:05:08.542301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.542164 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" Oct 30 06:05:08.542301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.542191 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" event={"ID":"2c0d1815-03a6-4485-86d0-c59b51432a58","Type":"ContainerDied","Data":"4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0"} Oct 30 06:05:08.542301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.542249 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z" event={"ID":"2c0d1815-03a6-4485-86d0-c59b51432a58","Type":"ContainerDied","Data":"9cc6706b0f2dd96b136bda935fbbad6fdccbc429244edfdd7efe07078224dcb2"} Oct 30 06:05:08.542301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.542281 5565 scope.go:122] "RemoveContainer" containerID="4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0" Oct 30 06:05:08.553899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.553873 5565 scope.go:122] "RemoveContainer" containerID="4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0" Oct 30 06:05:08.554265 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:08.554239 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0\": container with ID starting with 4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0 not found: ID does not exist" containerID="4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0" Oct 30 06:05:08.554351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.554276 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0"} err="failed to get container status \"4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0\": rpc error: code = NotFound desc = could not find container \"4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0\": container with ID starting with 4f9fcd63a9025b29dab760bdb4e2135a927d4183a4f9eff59ef3f0b38e5633d0 not found: ID does not exist" Oct 30 06:05:08.557873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.557848 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.557999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.557915 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.557999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.557984 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.558125 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.558017 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.558125 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.558111 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.558242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.558160 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.558242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.558216 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.558416 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.558340 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.558416 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.558400 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs\") pod \"2c0d1815-03a6-4485-86d0-c59b51432a58\" (UID: \"2c0d1815-03a6-4485-86d0-c59b51432a58\") " Oct 30 06:05:08.559137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.559105 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:08.559312 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.559287 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:08.559457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.559434 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c0d1815-03a6-4485-86d0-c59b51432a58-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.559682 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.559657 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:08.561347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.561325 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:08.561464 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.561375 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:08.561728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.561704 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:08.561728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.561717 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:08.563646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.563619 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "kube-api-access-xrbgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:08.564701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.564678 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo" pod "2c0d1815-03a6-4485-86d0-c59b51432a58" (UID: "2c0d1815-03a6-4485-86d0-c59b51432a58"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:08.661729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661700 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661728 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661739 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661749 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661762 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661774 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrbgd\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-kube-api-access-xrbgd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661783 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.661998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.661791 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c0d1815-03a6-4485-86d0-c59b51432a58-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:08.897507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.897470 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z"] Oct 30 06:05:08.912342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:08.912300 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-ct77z"] Oct 30 06:05:09.530148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:09.530109 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:05:09.530561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:09.530161 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:05:09.569599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:09.569569 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" path="/var/lib/kubelet/pods/2c0d1815-03a6-4485-86d0-c59b51432a58/volumes" Oct 30 06:05:09.570143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:09.570128 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" path="/var/lib/kubelet/pods/76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e/volumes" Oct 30 06:05:09.972397 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:09.972309 5565 patch_prober.go:29] interesting pod/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:05:09.972565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:09.972390 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:05:10.823578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.823550 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 06:05:10.900807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.900776 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.900874 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901336 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901382 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901417 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901464 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901491 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.901663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901599 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.902021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.901974 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:10.902115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.902060 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:10.902115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.902086 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:10.902230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.902092 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:10.902230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.902201 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket\") pod \"44d3ab58-857a-4d45-abdb-5e05d101895c\" (UID: \"44d3ab58-857a-4d45-abdb-5e05d101895c\") " Oct 30 06:05:10.902491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.902463 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:10.903159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.903134 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:10.903435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.903289 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44d3ab58-857a-4d45-abdb-5e05d101895c-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:10.903435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.903316 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:10.903435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.903315 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:10.903435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.903330 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:10.903435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.903343 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:10.904740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.904708 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "kube-api-access-24r9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:10.905309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.905284 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:10.907300 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:10.907270 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo" pod "44d3ab58-857a-4d45-abdb-5e05d101895c" (UID: "44d3ab58-857a-4d45-abdb-5e05d101895c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:11.021570 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.021510 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.021977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.021870 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.021977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.021926 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.021977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.021942 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24r9p\" (UniqueName: \"kubernetes.io/projected/44d3ab58-857a-4d45-abdb-5e05d101895c-kube-api-access-24r9p\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.087389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.087361 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 06:05:11.225177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225148 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.225328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225183 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.225328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225239 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.225328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225311 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225374 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225457 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225519 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225595 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225613 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225648 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225707 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225774 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.225896 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket\") pod \"56d49fe1-2070-4e60-90b3-4c077784c7ce\" (UID: \"56d49fe1-2070-4e60-90b3-4c077784c7ce\") " Oct 30 06:05:11.226218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.226166 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.226851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.226674 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.226851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.226690 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56d49fe1-2070-4e60-90b3-4c077784c7ce-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.226851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.226700 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.226851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.226709 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.226851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.226718 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.227756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.227736 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:11.227859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.227845 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:11.228119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.228099 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:11.228182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.228128 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw" pod "56d49fe1-2070-4e60-90b3-4c077784c7ce" (UID: "56d49fe1-2070-4e60-90b3-4c077784c7ce"). InnerVolumeSpecName "kube-api-access-98zgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:11.327984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.327951 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.327984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.327981 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98zgw\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-kube-api-access-98zgw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.327984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.327991 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.328291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.327999 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56d49fe1-2070-4e60-90b3-4c077784c7ce-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:11.572302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.572264 5565 generic.go:356] "Generic (PLEG): container finished" podID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerID="e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1" exitCode=0 Oct 30 06:05:11.572495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.572309 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" Oct 30 06:05:11.572495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.572345 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" event={"ID":"56d49fe1-2070-4e60-90b3-4c077784c7ce","Type":"ContainerDied","Data":"e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1"} Oct 30 06:05:11.572495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.572399 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2" event={"ID":"56d49fe1-2070-4e60-90b3-4c077784c7ce","Type":"ContainerDied","Data":"fb150b7b12a1de878d076714db84478e146805b1ac78e3e9d7a91bf27073903c"} Oct 30 06:05:11.572495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.572423 5565 scope.go:122] "RemoveContainer" containerID="e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1" Oct 30 06:05:11.574315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.574292 5565 generic.go:356] "Generic (PLEG): container finished" podID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerID="86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5" exitCode=0 Oct 30 06:05:11.574429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.574316 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" event={"ID":"44d3ab58-857a-4d45-abdb-5e05d101895c","Type":"ContainerDied","Data":"86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5"} Oct 30 06:05:11.574429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.574363 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" event={"ID":"44d3ab58-857a-4d45-abdb-5e05d101895c","Type":"ContainerDied","Data":"c9b9dfeefa3ecc8359c8025068a10104e0a2e2993d251d75b73f9c102689a119"} Oct 30 06:05:11.574429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.574413 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m" Oct 30 06:05:11.586294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.586272 5565 scope.go:122] "RemoveContainer" containerID="e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1" Oct 30 06:05:11.586644 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:11.586617 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1\": container with ID starting with e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1 not found: ID does not exist" containerID="e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1" Oct 30 06:05:11.586732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.586650 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1"} err="failed to get container status \"e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1\": rpc error: code = NotFound desc = could not find container \"e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1\": container with ID starting with e8d6b0d535989e3b2bdb361c85e49bf54058803a1c490d0b46f6b79130c1b5b1 not found: ID does not exist" Oct 30 06:05:11.586732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.586669 5565 scope.go:122] "RemoveContainer" containerID="86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5" Oct 30 06:05:11.596850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.596835 5565 scope.go:122] "RemoveContainer" containerID="86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5" Oct 30 06:05:11.597138 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:11.597122 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5\": container with ID starting with 86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5 not found: ID does not exist" containerID="86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5" Oct 30 06:05:11.597186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.597150 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5"} err="failed to get container status \"86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5\": rpc error: code = NotFound desc = could not find container \"86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5\": container with ID starting with 86c58bb2864cdf57d1ab22311f38d00a70eb5d6ada0470a66358de3fe72bb1f5 not found: ID does not exist" Oct 30 06:05:11.635058 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.635023 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m"] Oct 30 06:05:11.649493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.649459 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-r6q4m"] Oct 30 06:05:11.680736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.680704 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2"] Oct 30 06:05:11.686408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:11.686382 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-8blt2"] Oct 30 06:05:12.525720 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.525696 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 06:05:12.587493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.587457 5565 generic.go:356] "Generic (PLEG): container finished" podID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerID="b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692" exitCode=0 Oct 30 06:05:12.587731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.587520 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" event={"ID":"1d56599e-90f7-4183-a8d8-dff62f249f3b","Type":"ContainerDied","Data":"b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692"} Oct 30 06:05:12.587731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.587590 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" event={"ID":"1d56599e-90f7-4183-a8d8-dff62f249f3b","Type":"ContainerDied","Data":"3c80d0bf42a40cd517a037b1630ce96bf58869eade2072e8e628ec83824edee1"} Oct 30 06:05:12.587731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.587615 5565 scope.go:122] "RemoveContainer" containerID="b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692" Oct 30 06:05:12.587731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.587706 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf" Oct 30 06:05:12.607372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.607342 5565 scope.go:122] "RemoveContainer" containerID="b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692" Oct 30 06:05:12.607837 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:12.607800 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692\": container with ID starting with b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692 not found: ID does not exist" containerID="b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692" Oct 30 06:05:12.607946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.607844 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692"} err="failed to get container status \"b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692\": rpc error: code = NotFound desc = could not find container \"b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692\": container with ID starting with b609a88548346bef8e223f5f1e644358384702a0e506b68b691bffbc6dc0b692 not found: ID does not exist" Oct 30 06:05:12.647805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.647762 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.647999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.647883 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.647999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.647915 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.647999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.647978 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.648183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.648157 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.648323 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.648243 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.648422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.648372 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.648480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.648430 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.648480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.648461 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data\") pod \"1d56599e-90f7-4183-a8d8-dff62f249f3b\" (UID: \"1d56599e-90f7-4183-a8d8-dff62f249f3b\") " Oct 30 06:05:12.648646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.648623 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:12.649485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.649391 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:12.649485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.649474 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.649865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.649807 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:12.649865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.649831 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:12.650119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.650098 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:12.652905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.652883 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:12.653016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.652901 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:12.654633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.654608 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:12.654726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.654644 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx" pod "1d56599e-90f7-4183-a8d8-dff62f249f3b" (UID: "1d56599e-90f7-4183-a8d8-dff62f249f3b"). InnerVolumeSpecName "kube-api-access-zsxlx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:12.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750784 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750814 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750836 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsxlx\" (UniqueName: \"kubernetes.io/projected/1d56599e-90f7-4183-a8d8-dff62f249f3b-kube-api-access-zsxlx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750845 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d56599e-90f7-4183-a8d8-dff62f249f3b-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750854 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.750864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750863 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.751220 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750873 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.751220 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.750882 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d56599e-90f7-4183-a8d8-dff62f249f3b-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:12.963054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.963022 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf"] Oct 30 06:05:12.998982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:12.998950 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-787sf"] Oct 30 06:05:13.570162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.570132 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" path="/var/lib/kubelet/pods/1d56599e-90f7-4183-a8d8-dff62f249f3b/volumes" Oct 30 06:05:13.570670 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.570654 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" path="/var/lib/kubelet/pods/44d3ab58-857a-4d45-abdb-5e05d101895c/volumes" Oct 30 06:05:13.571129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.571116 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" path="/var/lib/kubelet/pods/56d49fe1-2070-4e60-90b3-4c077784c7ce/volumes" Oct 30 06:05:13.673049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673013 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl"] Oct 30 06:05:13.673804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673787 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" Oct 30 06:05:13.673804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673804 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673823 5565 cpu_manager.go:414] "containerMap: removing container" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673829 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673847 5565 cpu_manager.go:414] "containerMap: removing container" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673853 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673858 5565 cpu_manager.go:414] "containerMap: removing container" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673863 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673875 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" Oct 30 06:05:13.673939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.673879 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" Oct 30 06:05:13.674180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.674007 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d56599e-90f7-4183-a8d8-dff62f249f3b" containerName="istio-proxy" Oct 30 06:05:13.674180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.674015 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="44d3ab58-857a-4d45-abdb-5e05d101895c" containerName="istio-proxy" Oct 30 06:05:13.674180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.674024 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="56d49fe1-2070-4e60-90b3-4c077784c7ce" containerName="istio-proxy" Oct 30 06:05:13.674180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.674031 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" Oct 30 06:05:13.674180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.674039 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c0d1815-03a6-4485-86d0-c59b51432a58" containerName="istio-proxy" Oct 30 06:05:13.674180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.674045 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" Oct 30 06:05:13.679129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.679112 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.712190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.712163 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-7dfcb7-openshift-default-dockercfg-h8vd6\"" Oct 30 06:05:13.719008 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.718981 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl"] Oct 30 06:05:13.764938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.764911 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.764983 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765143 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxl29\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765174 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765252 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765348 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765288 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765429 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765481 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.765581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.765557 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.866958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.866887 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxl29\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.866958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.866926 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867131 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.866975 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867131 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867023 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867142 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867209 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867243 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867326 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867384 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867690 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867755 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867763 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.867803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.867796 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.868235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.868209 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.869634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.869611 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.869736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.869695 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.907606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.907584 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxl29\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.907909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.907888 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token\") pod \"e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:13.993027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:13.993003 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:14.322701 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:05:14.322674 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a58a055_136e_4160_b118_b26a3674b80f.slice/crio-4d8405edee82b974a2c6b5470638f3762ce79bcbb908f2d0b2752daf4f2986b5 WatchSource:0}: Error finding container 4d8405edee82b974a2c6b5470638f3762ce79bcbb908f2d0b2752daf4f2986b5: Status 404 returned error can't find the container with id 4d8405edee82b974a2c6b5470638f3762ce79bcbb908f2d0b2752daf4f2986b5 Oct 30 06:05:14.323685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.323572 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl"] Oct 30 06:05:14.325935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.325905 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:05:14.326017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.325988 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:05:14.326065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.326028 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:05:14.605671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.605560 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" event={"ID":"2a58a055-136e-4160-b118-b26a3674b80f","Type":"ContainerStarted","Data":"202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d"} Oct 30 06:05:14.605671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.605628 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" event={"ID":"2a58a055-136e-4160-b118-b26a3674b80f","Type":"ContainerStarted","Data":"4d8405edee82b974a2c6b5470638f3762ce79bcbb908f2d0b2752daf4f2986b5"} Oct 30 06:05:14.687208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.687163 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podStartSLOduration=1.687149765 podStartE2EDuration="1.687149765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:14.68508946 +0000 UTC m=+2021.260776514" watchObservedRunningTime="2026-10-30 06:05:14.687149765 +0000 UTC m=+2021.262836818" Oct 30 06:05:14.994069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:14.993990 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:15.995657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:15.995571 5565 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 06:05:15.995657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:15.995622 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 06:05:16.993582 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:16.993508 5565 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 06:05:16.993742 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:16.993606 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 06:05:17.539897 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:17.539859 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:05:17.993342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:17.993256 5565 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 06:05:17.993342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:17.993317 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 06:05:18.994040 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:18.993999 5565 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 06:05:18.994591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:18.994076 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 06:05:19.994028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:19.993996 5565 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 06:05:19.994207 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:19.994059 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 06:05:20.649986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.649949 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz"] Oct 30 06:05:20.650819 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.650797 5565 cpu_manager.go:414] "containerMap: removing container" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" Oct 30 06:05:20.650819 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.650815 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76bc5ae8-9e34-4b7e-a882-ccf1f28ecc4e" containerName="istio-proxy" Oct 30 06:05:20.655582 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.655559 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.679887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.679855 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz"] Oct 30 06:05:20.687641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.687618 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-38cfad-openshift-default-dockercfg-28vsq\"" Oct 30 06:05:20.815152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815119 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815284 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bj95\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815317 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815556 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815637 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815701 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815747 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815772 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.815826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.815804 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.851951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.851920 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7"] Oct 30 06:05:20.852150 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.852126 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" containerID="cri-o://2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975" gracePeriod=30 Oct 30 06:05:20.917576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917441 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.917576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917503 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.917782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917626 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.917782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917717 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.917782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917751 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.917886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917804 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.917957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.917939 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.918060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.918039 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.918103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.918061 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bj95\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.918103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.918097 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.918359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.918339 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.918689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.918667 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.918841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.918725 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.919053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.919030 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.920575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.920555 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.920999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.920978 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.944422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.944396 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bj95\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:20.944696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:20.944675 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token\") pod \"e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:21.000425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.000385 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:21.000950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.000780 5565 patch_prober.go:29] interesting pod/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" start-of-body= Oct 30 06:05:21.000950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.000846 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.209:15021/healthz/ready\": dial tcp 10.134.0.209:15021: connect: connection refused" Oct 30 06:05:21.353932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.353900 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz"] Oct 30 06:05:21.354311 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:05:21.354288 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a96ca1a_bfd3_42dd_a2a2_718bfc1eb891.slice/crio-b36bbc80327191f9016872b3d62278e55f9dda1d6e24a516b370d7b06c268b7a WatchSource:0}: Error finding container b36bbc80327191f9016872b3d62278e55f9dda1d6e24a516b370d7b06c268b7a: Status 404 returned error can't find the container with id b36bbc80327191f9016872b3d62278e55f9dda1d6e24a516b370d7b06c268b7a Oct 30 06:05:21.357103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.357073 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:05:21.357175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.357134 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:05:21.357175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.357166 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:05:21.672634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.670997 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" event={"ID":"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891","Type":"ContainerStarted","Data":"dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8"} Oct 30 06:05:21.672634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.671064 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" event={"ID":"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891","Type":"ContainerStarted","Data":"b36bbc80327191f9016872b3d62278e55f9dda1d6e24a516b370d7b06c268b7a"} Oct 30 06:05:21.735150 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.735097 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podStartSLOduration=1.735082889 podStartE2EDuration="1.735082889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:21.730760613 +0000 UTC m=+2028.306447666" watchObservedRunningTime="2026-10-30 06:05:21.735082889 +0000 UTC m=+2028.310769942" Oct 30 06:05:21.999138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.999067 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:21.999588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:21.999569 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:22.000138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:22.000118 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:05:22.001217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:22.001197 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:22.002473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:22.002450 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:22.002593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:22.002496 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:22.493629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:22.493393 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t"] Oct 30 06:05:23.001058 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:23.001021 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:23.001234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:23.001079 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:24.001302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:24.001270 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:24.001771 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:24.001323 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:25.001406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:25.001371 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:25.001827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:25.001423 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:26.002420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.001467 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:26.002420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.001541 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:26.134802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.134737 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 06:05:26.138602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138578 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.138728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138636 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.138773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138739 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.138810 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138771 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.138869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138847 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.138933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138914 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.138999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138966 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.139055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.138991 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:05:26.139055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139039 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.139171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139089 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.139238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139170 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7\") pod \"f538d252-0856-4886-823a-1cd114d0f319\" (UID: \"f538d252-0856-4886-823a-1cd114d0f319\") " Oct 30 06:05:26.139753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139729 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f538d252-0856-4886-823a-1cd114d0f319-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.139872 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139755 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.139872 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139780 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.139872 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139809 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.140032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.139940 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.141476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.141430 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:26.141646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.141599 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:26.141927 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.141905 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "kube-api-access-vnqm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:26.142328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.142309 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo" pod "f538d252-0856-4886-823a-1cd114d0f319" (UID: "f538d252-0856-4886-823a-1cd114d0f319"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:05:26.242073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242028 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.242073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242064 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.242073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242079 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnqm7\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-kube-api-access-vnqm7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.242378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242093 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f538d252-0856-4886-823a-1cd114d0f319-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.242378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242107 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.242378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242119 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f538d252-0856-4886-823a-1cd114d0f319-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.242378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.242131 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f538d252-0856-4886-823a-1cd114d0f319-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:26.851192 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.851159 5565 generic.go:356] "Generic (PLEG): container finished" podID="f538d252-0856-4886-823a-1cd114d0f319" containerID="2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975" exitCode=0 Oct 30 06:05:26.851379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.851215 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" Oct 30 06:05:26.851379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.851242 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" event={"ID":"f538d252-0856-4886-823a-1cd114d0f319","Type":"ContainerDied","Data":"2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975"} Oct 30 06:05:26.851379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.851302 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7" event={"ID":"f538d252-0856-4886-823a-1cd114d0f319","Type":"ContainerDied","Data":"b93c85df77874f7e590aa6cc93d47b3361881d0babe9e30c2391f82b74b786e3"} Oct 30 06:05:26.851379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.851331 5565 scope.go:122] "RemoveContainer" containerID="2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975" Oct 30 06:05:26.885521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.885501 5565 scope.go:122] "RemoveContainer" containerID="2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975" Oct 30 06:05:26.885856 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:26.885831 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975\": container with ID starting with 2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975 not found: ID does not exist" containerID="2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975" Oct 30 06:05:26.885899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.885863 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975"} err="failed to get container status \"2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975\": rpc error: code = NotFound desc = could not find container \"2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975\": container with ID starting with 2efad19474e3116f37a32de1b5a945a711ccfe02efca1e0cea7fb8ac72462975 not found: ID does not exist" Oct 30 06:05:26.927303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.927276 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7"] Oct 30 06:05:26.941109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:26.941088 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-mlsr7"] Oct 30 06:05:27.001038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:27.001003 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:27.001204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:27.001064 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:27.596839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:27.596806 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f538d252-0856-4886-823a-1cd114d0f319" path="/var/lib/kubelet/pods/f538d252-0856-4886-823a-1cd114d0f319/volumes" Oct 30 06:05:28.001181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:28.001093 5565 patch_prober.go:29] interesting pod/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" start-of-body= Oct 30 06:05:28.001181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:28.001168 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.210:15021/healthz/ready\": dial tcp 10.134.0.210:15021: connect: connection refused" Oct 30 06:05:29.007131 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:29.007096 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:29.007567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:29.007552 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:29.008295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:29.008272 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:05:37.518764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.518692 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" podUID="04b05d01-a811-4f7a-9ab1-012db09e0a93" containerName="main" containerID="cri-o://4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf" gracePeriod=45 Oct 30 06:05:37.807887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.807857 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:05:37.944615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.944580 5565 scope.go:122] "RemoveContainer" containerID="243f5b3ef2c0628913403f5a86d8ae751ea467794382040f52336ad772908c3d" Oct 30 06:05:37.985106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985084 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm\") pod \"04b05d01-a811-4f7a-9ab1-012db09e0a93\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " Oct 30 06:05:37.985189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985125 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml\" (UniqueName: \"kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml\") pod \"04b05d01-a811-4f7a-9ab1-012db09e0a93\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " Oct 30 06:05:37.985189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985162 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home\") pod \"04b05d01-a811-4f7a-9ab1-012db09e0a93\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " Oct 30 06:05:37.985275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985201 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir\") pod \"04b05d01-a811-4f7a-9ab1-012db09e0a93\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " Oct 30 06:05:37.985424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985404 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs\" (UniqueName: \"kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs\") pod \"04b05d01-a811-4f7a-9ab1-012db09e0a93\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " Oct 30 06:05:37.985506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985444 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home" pod "04b05d01-a811-4f7a-9ab1-012db09e0a93" (UID: "04b05d01-a811-4f7a-9ab1-012db09e0a93"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:37.985595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985501 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache\") pod \"04b05d01-a811-4f7a-9ab1-012db09e0a93\" (UID: \"04b05d01-a811-4f7a-9ab1-012db09e0a93\") " Oct 30 06:05:37.985595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985510 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir" pod "04b05d01-a811-4f7a-9ab1-012db09e0a93" (UID: "04b05d01-a811-4f7a-9ab1-012db09e0a93"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:37.985767 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.985736 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache" pod "04b05d01-a811-4f7a-9ab1-012db09e0a93" (UID: "04b05d01-a811-4f7a-9ab1-012db09e0a93"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:37.986250 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.986231 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:37.986250 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.986252 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:37.986402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.986260 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:37.987480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.987460 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm" pod "04b05d01-a811-4f7a-9ab1-012db09e0a93" (UID: "04b05d01-a811-4f7a-9ab1-012db09e0a93"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:05:37.987587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.987545 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs" pod "04b05d01-a811-4f7a-9ab1-012db09e0a93" (UID: "04b05d01-a811-4f7a-9ab1-012db09e0a93"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:05:37.987992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:37.987972 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml" pod "04b05d01-a811-4f7a-9ab1-012db09e0a93" (UID: "04b05d01-a811-4f7a-9ab1-012db09e0a93"). InnerVolumeSpecName "kube-api-access-qjsml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:05:38.065357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.065280 5565 generic.go:356] "Generic (PLEG): container finished" podID="04b05d01-a811-4f7a-9ab1-012db09e0a93" containerID="4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf" exitCode=0 Oct 30 06:05:38.065357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.065325 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" Oct 30 06:05:38.065357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.065344 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" event={"ID":"04b05d01-a811-4f7a-9ab1-012db09e0a93","Type":"ContainerDied","Data":"4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf"} Oct 30 06:05:38.065588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.065378 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t" event={"ID":"04b05d01-a811-4f7a-9ab1-012db09e0a93","Type":"ContainerDied","Data":"be9e80b13e1dfaa8bb052c33da8fc2ec4da227684304469edb676cda99371a5f"} Oct 30 06:05:38.065588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.065396 5565 scope.go:122] "RemoveContainer" containerID="4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf" Oct 30 06:05:38.076089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.076071 5565 scope.go:122] "RemoveContainer" containerID="4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf" Oct 30 06:05:38.076362 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:38.076340 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf\": container with ID starting with 4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf not found: ID does not exist" containerID="4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf" Oct 30 06:05:38.076451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.076368 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf"} err="failed to get container status \"4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf\": rpc error: code = NotFound desc = could not find container \"4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf\": container with ID starting with 4a577045606d697f3f9ac39ed135983caa5170faaffbc5c7cb32dc4ff2ab5cbf not found: ID does not exist" Oct 30 06:05:38.122550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.111017 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04b05d01-a811-4f7a-9ab1-012db09e0a93-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:38.122550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.111055 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/04b05d01-a811-4f7a-9ab1-012db09e0a93-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:38.122550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.111072 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qjsml\" (UniqueName: \"kubernetes.io/projected/04b05d01-a811-4f7a-9ab1-012db09e0a93-kube-api-access-qjsml\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:05:38.178548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.177516 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t"] Oct 30 06:05:38.208552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:38.198440 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-765de9/test-model-765de9-kserve-74c4989c8c-fwk2t"] Oct 30 06:05:39.569799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:39.569764 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b05d01-a811-4f7a-9ab1-012db09e0a93" path="/var/lib/kubelet/pods/04b05d01-a811-4f7a-9ab1-012db09e0a93/volumes" Oct 30 06:05:47.536034 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:05:47.534289 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:05:48.816256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.816221 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5"] Oct 30 06:05:48.817042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.817025 5565 cpu_manager.go:414] "containerMap: removing container" podUID="04b05d01-a811-4f7a-9ab1-012db09e0a93" containerName="main" Oct 30 06:05:48.817042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.817043 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b05d01-a811-4f7a-9ab1-012db09e0a93" containerName="main" Oct 30 06:05:48.817132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.817074 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" Oct 30 06:05:48.817132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.817079 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" Oct 30 06:05:48.817216 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.817205 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b05d01-a811-4f7a-9ab1-012db09e0a93" containerName="main" Oct 30 06:05:48.817251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.817221 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f538d252-0856-4886-823a-1cd114d0f319" containerName="istio-proxy" Oct 30 06:05:48.822915 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.822894 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:48.850896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.850867 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5"] Oct 30 06:05:48.860050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.860028 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-6b8f87\"" Oct 30 06:05:48.917783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.917750 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd"] Oct 30 06:05:48.922705 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.922682 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:48.930160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.930128 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:48.930332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.930261 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-642v8\" (UniqueName: \"kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8\") pod \"payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:48.961607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:48.958176 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-6b8f87-dockercfg-5jhvr\"" Oct 30 06:05:49.062556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.060880 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:49.062556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.060955 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-642v8\" (UniqueName: \"kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8\") pod \"payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:49.062556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.061220 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95qz7\" (UniqueName: \"kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7\") pod \"payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.062556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.061381 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.108600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.104573 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:49.200686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.200653 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-642v8\" (UniqueName: \"kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8\") pod \"payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:49.206274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.204483 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95qz7\" (UniqueName: \"kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7\") pod \"payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.206918 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.204904 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:49.207115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.207094 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.208744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.208722 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.290367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.270855 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd"] Oct 30 06:05:49.290367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.274271 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95qz7\" (UniqueName: \"kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7\") pod \"payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.553579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.553542 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:49.620161 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:05:49.620128 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2795ff00_a5e2_4bb0_baa7_70a86a50429d.slice/crio-71eceb1b156069057b17ef4fda48289598e19a4526f0980f25d7e613a0ed007e WatchSource:0}: Error finding container 71eceb1b156069057b17ef4fda48289598e19a4526f0980f25d7e613a0ed007e: Status 404 returned error can't find the container with id 71eceb1b156069057b17ef4fda48289598e19a4526f0980f25d7e613a0ed007e Oct 30 06:05:49.621196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.621114 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5"] Oct 30 06:05:49.867905 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:05:49.867855 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod820c0009_c7e3_4035_a4ba_733b27571362.slice/crio-fccff1111c8fb5e3ffcc1b77e6f6c696899b12c4d3db9eddac3d26869e1fa672 WatchSource:0}: Error finding container fccff1111c8fb5e3ffcc1b77e6f6c696899b12c4d3db9eddac3d26869e1fa672: Status 404 returned error can't find the container with id fccff1111c8fb5e3ffcc1b77e6f6c696899b12c4d3db9eddac3d26869e1fa672 Oct 30 06:05:49.868310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:49.868289 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd"] Oct 30 06:05:50.212597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.212499 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" event={"ID":"820c0009-c7e3-4035-a4ba-733b27571362","Type":"ContainerStarted","Data":"93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e"} Oct 30 06:05:50.212597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.212578 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:05:50.212789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.212607 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" event={"ID":"820c0009-c7e3-4035-a4ba-733b27571362","Type":"ContainerStarted","Data":"fccff1111c8fb5e3ffcc1b77e6f6c696899b12c4d3db9eddac3d26869e1fa672"} Oct 30 06:05:50.214086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.214059 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" event={"ID":"2795ff00-a5e2-4bb0-baa7-70a86a50429d","Type":"ContainerStarted","Data":"5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b"} Oct 30 06:05:50.214244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.214105 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" event={"ID":"2795ff00-a5e2-4bb0-baa7-70a86a50429d","Type":"ContainerStarted","Data":"71eceb1b156069057b17ef4fda48289598e19a4526f0980f25d7e613a0ed007e"} Oct 30 06:05:50.214692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.214668 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:05:50.258696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.258648 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" podStartSLOduration=2.258634685 podStartE2EDuration="2.258634685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:50.256723621 +0000 UTC m=+2056.832410676" watchObservedRunningTime="2026-10-30 06:05:50.258634685 +0000 UTC m=+2056.834321808" Oct 30 06:05:50.303808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:05:50.303770 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" podStartSLOduration=2.303757608 podStartE2EDuration="2.303757608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:05:50.301042961 +0000 UTC m=+2056.876730020" watchObservedRunningTime="2026-10-30 06:05:50.303757608 +0000 UTC m=+2056.879444661" Oct 30 06:06:01.226769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:01.226737 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:06:01.227276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:01.226819 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:06:05.033016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.032963 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd"] Oct 30 06:06:05.048700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.048662 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bc288054-g57gd"] Oct 30 06:06:05.703547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.702696 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5546c62-7dcd-47a4-a3f4-9fa1ca646b04" path="/var/lib/kubelet/pods/b5546c62-7dcd-47a4-a3f4-9fa1ca646b04/volumes" Oct 30 06:06:05.881152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.881100 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg"] Oct 30 06:06:05.887765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.887734 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:05.889661 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.889624 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg"] Oct 30 06:06:05.910003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.909970 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:05.922281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.922250 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:06:05.944907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.944863 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:05.945104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.944943 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f5sk\" (UniqueName: \"kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:05.945269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:05.945239 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.046539 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.046504 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.047068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.046795 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.047068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.046838 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f5sk\" (UniqueName: \"kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.047664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.047634 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.049326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.049302 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.083867 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.083834 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f5sk\" (UniqueName: \"kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk\") pod \"maas-api-revoke-keys-e2e-inf-b-765de9-g46hg\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.215618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.215573 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:06.582516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:06.582486 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg"] Oct 30 06:06:06.584806 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:06:06.584781 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246fbf1b_0fc1_4f2b_bfb2_f7d2dd0390d7.slice/crio-543c2f6e64ca4b90ccd9a114f6c2d20299d39629095424272996485a89e19e26 WatchSource:0}: Error finding container 543c2f6e64ca4b90ccd9a114f6c2d20299d39629095424272996485a89e19e26: Status 404 returned error can't find the container with id 543c2f6e64ca4b90ccd9a114f6c2d20299d39629095424272996485a89e19e26 Oct 30 06:06:07.524898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:07.524858 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" event={"ID":"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7","Type":"ContainerStarted","Data":"fd44dfdfe1ca50afd29043eb5f87f2c1cb880b26d6834a77d13b1a55fd9cec3e"} Oct 30 06:06:07.525339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:07.524928 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" event={"ID":"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7","Type":"ContainerStarted","Data":"543c2f6e64ca4b90ccd9a114f6c2d20299d39629095424272996485a89e19e26"} Oct 30 06:06:07.574600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:07.574488 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" podStartSLOduration=2.5744719480000002 podStartE2EDuration="2.574471948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:07.570377832 +0000 UTC m=+2074.146064887" watchObservedRunningTime="2026-10-30 06:06:07.574471948 +0000 UTC m=+2074.150159006" Oct 30 06:06:08.535167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:08.535123 5565 generic.go:356] "Generic (PLEG): container finished" podID="246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" containerID="fd44dfdfe1ca50afd29043eb5f87f2c1cb880b26d6834a77d13b1a55fd9cec3e" exitCode=0 Oct 30 06:06:08.535637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:08.535210 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" event={"ID":"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7","Type":"ContainerDied","Data":"fd44dfdfe1ca50afd29043eb5f87f2c1cb880b26d6834a77d13b1a55fd9cec3e"} Oct 30 06:06:10.209138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.209103 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:10.314100 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.313991 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk\" (UniqueName: \"kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk\") pod \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " Oct 30 06:06:10.314288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.314265 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca\") pod \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " Oct 30 06:06:10.314435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.314416 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca\") pod \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\" (UID: \"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7\") " Oct 30 06:06:10.314812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.314770 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca" pod "246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" (UID: "246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:10.315401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.315372 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.367807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.367750 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca" pod "246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" (UID: "246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:10.367978 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.367865 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk" pod "246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" (UID: "246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7"). InnerVolumeSpecName "kube-api-access-4f5sk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:10.417109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.417070 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.417109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.417105 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f5sk\" (UniqueName: \"kubernetes.io/projected/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7-kube-api-access-4f5sk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:10.565640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.565551 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" Oct 30 06:06:10.565885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.565839 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg" event={"ID":"246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7","Type":"ContainerDied","Data":"543c2f6e64ca4b90ccd9a114f6c2d20299d39629095424272996485a89e19e26"} Oct 30 06:06:10.566007 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:10.565913 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="543c2f6e64ca4b90ccd9a114f6c2d20299d39629095424272996485a89e19e26" Oct 30 06:06:16.877221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:16.877167 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": read tcp 10.134.0.2:53828->10.134.0.206:15021: read: connection reset by peer" start-of-body= Oct 30 06:06:16.882727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:16.877751 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": read tcp 10.134.0.2:53828->10.134.0.206:15021: read: connection reset by peer" Oct 30 06:06:17.536155 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:06:17.536117 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:06:17.634347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:17.634308 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerID="31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472" exitCode=0 Oct 30 06:06:17.634571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:17.634378 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" event={"ID":"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea","Type":"ContainerDied","Data":"31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472"} Oct 30 06:06:17.634929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:17.634912 5565 scope.go:122] "RemoveContainer" containerID="31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472" Oct 30 06:06:17.637018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:17.636988 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:17.637113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:17.637060 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:17.637113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:17.637100 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:18.645120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:18.645083 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" event={"ID":"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea","Type":"ContainerStarted","Data":"4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a"} Oct 30 06:06:19.032493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:19.032462 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7"] Oct 30 06:06:19.038589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:19.038561 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2c91a764-dvcj7"] Oct 30 06:06:19.159555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:19.153816 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:06:19.159761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:19.159641 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 06:06:19.159761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:19.159700 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 06:06:19.610552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:19.609577 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d4dc6c-a13f-419f-9df2-977e50e6e452" path="/var/lib/kubelet/pods/d0d4dc6c-a13f-419f-9df2-977e50e6e452/volumes" Oct 30 06:06:20.172550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:20.156844 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 06:06:20.172550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:20.156914 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 06:06:21.179903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.179866 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 06:06:21.180581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.180505 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 06:06:21.589201 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.589172 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p"] Oct 30 06:06:21.589976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.589959 5565 cpu_manager.go:414] "containerMap: removing container" podUID="246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" containerName="revoke-keys" Oct 30 06:06:21.590031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.589978 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" containerName="revoke-keys" Oct 30 06:06:21.590126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.590117 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" containerName="revoke-keys" Oct 30 06:06:21.593023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.593001 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p"] Oct 30 06:06:21.593193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.593086 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.609839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.609813 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:21.623806 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.623784 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:06:21.704387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.704350 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.704593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.704423 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-99ppn\" (UniqueName: \"kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.704667 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.704647 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.801458 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.801411 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv"] Oct 30 06:06:21.805301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.805278 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.806273 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.806243 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.806451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.806432 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-99ppn\" (UniqueName: \"kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.806698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.806680 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.807079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.807056 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.809001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.808980 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.825481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.825459 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1dbaf6-openshift-default-dockercfg-vtk9c\"" Oct 30 06:06:21.830126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.830102 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv"] Oct 30 06:06:21.833066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.833038 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-99ppn\" (UniqueName: \"kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn\") pod \"maas-api-revoke-sub-b942db37-r5z4p\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:21.908095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.907992 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908046 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908203 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908260 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908319 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908341 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908375 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpsdc\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908566 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.908757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.908658 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:21.911866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:21.911847 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:22.010775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.010739 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.011928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.011806 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012038 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012100 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012148 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012163 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012196 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012214 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012246 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpsdc\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012644 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.012902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.012876 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.013164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.013137 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.013447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.013187 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.013569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.013367 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.013643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.013596 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.015371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.015349 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.038836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.037867 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.038836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.038310 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpsdc\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc\") pod \"e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.136139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.136103 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:22.184376 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.183974 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 06:06:22.184376 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.184036 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 06:06:22.302382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.302340 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p"] Oct 30 06:06:22.554438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.554002 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv"] Oct 30 06:06:22.581896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.581163 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:22.581896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.581259 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:22.581896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.581299 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:22.709593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.709555 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" event={"ID":"e13de906-e6b8-4b42-a065-d6b6540d0e24","Type":"ContainerStarted","Data":"e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d"} Oct 30 06:06:22.709737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.709622 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" event={"ID":"e13de906-e6b8-4b42-a065-d6b6540d0e24","Type":"ContainerStarted","Data":"f0fc8ecbd94b848debc16ec13dd28a28473b8212f4725f64cba84e7e7c2dce34"} Oct 30 06:06:22.711303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.711274 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" event={"ID":"c959a373-cca2-4f23-bac1-05b1c4eba284","Type":"ContainerStarted","Data":"14d8131e8785e8d4eed9d1b5f0b72eb6585f988bc0da8ba084c8679361f7074a"} Oct 30 06:06:22.789577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:22.770736 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podStartSLOduration=1.770716342 podStartE2EDuration="1.770716342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:22.769859381 +0000 UTC m=+2089.345546435" watchObservedRunningTime="2026-10-30 06:06:22.770716342 +0000 UTC m=+2089.346403398" Oct 30 06:06:23.137101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.137065 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:23.138152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.138081 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:23.138152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.138127 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:23.175595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.157850 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 06:06:23.175595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.157905 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 06:06:23.721869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.721826 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" event={"ID":"c959a373-cca2-4f23-bac1-05b1c4eba284","Type":"ContainerStarted","Data":"21946dddfdf15c993773ecef58f7c9611ce01b11a9a919c31be052804ab6ee5e"} Oct 30 06:06:23.781816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:23.781765 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" podStartSLOduration=2.781752107 podStartE2EDuration="2.781752107s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:23.767896131 +0000 UTC m=+2090.343583180" watchObservedRunningTime="2026-10-30 06:06:23.781752107 +0000 UTC m=+2090.357439160" Oct 30 06:06:24.137356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:24.137321 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:24.137654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:24.137383 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:24.164644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:24.164609 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" start-of-body= Oct 30 06:06:24.164879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:24.164854 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.206:15021/healthz/ready\": dial tcp 10.134.0.206:15021: connect: connection refused" Oct 30 06:06:24.731929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:24.731889 5565 generic.go:356] "Generic (PLEG): container finished" podID="c959a373-cca2-4f23-bac1-05b1c4eba284" containerID="21946dddfdf15c993773ecef58f7c9611ce01b11a9a919c31be052804ab6ee5e" exitCode=0 Oct 30 06:06:24.732446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:24.731971 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" event={"ID":"c959a373-cca2-4f23-bac1-05b1c4eba284","Type":"ContainerDied","Data":"21946dddfdf15c993773ecef58f7c9611ce01b11a9a919c31be052804ab6ee5e"} Oct 30 06:06:25.137604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:25.136747 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:25.137604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:25.136810 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:25.179365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:25.179332 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:06:25.179959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:25.179935 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:06:25.181240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:25.181220 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:06:25.945324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:25.945296 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:26.040068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.040034 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca\") pod \"c959a373-cca2-4f23-bac1-05b1c4eba284\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " Oct 30 06:06:26.040244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.040215 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn\" (UniqueName: \"kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn\") pod \"c959a373-cca2-4f23-bac1-05b1c4eba284\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " Oct 30 06:06:26.040304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.040242 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca\") pod \"c959a373-cca2-4f23-bac1-05b1c4eba284\" (UID: \"c959a373-cca2-4f23-bac1-05b1c4eba284\") " Oct 30 06:06:26.040380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.040364 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca" pod "c959a373-cca2-4f23-bac1-05b1c4eba284" (UID: "c959a373-cca2-4f23-bac1-05b1c4eba284"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:26.040834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.040811 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:26.042490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.042465 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca" pod "c959a373-cca2-4f23-bac1-05b1c4eba284" (UID: "c959a373-cca2-4f23-bac1-05b1c4eba284"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:26.043631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.043603 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn" pod "c959a373-cca2-4f23-bac1-05b1c4eba284" (UID: "c959a373-cca2-4f23-bac1-05b1c4eba284"). InnerVolumeSpecName "kube-api-access-99ppn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:26.137115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.137086 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:26.137332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.137306 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:26.142106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.142079 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-99ppn\" (UniqueName: \"kubernetes.io/projected/c959a373-cca2-4f23-bac1-05b1c4eba284-kube-api-access-99ppn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:26.142106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.142106 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c959a373-cca2-4f23-bac1-05b1c4eba284-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:26.791021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.790969 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" Oct 30 06:06:26.794588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.791376 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p" event={"ID":"c959a373-cca2-4f23-bac1-05b1c4eba284","Type":"ContainerDied","Data":"14d8131e8785e8d4eed9d1b5f0b72eb6585f988bc0da8ba084c8679361f7074a"} Oct 30 06:06:26.794961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:26.794939 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="14d8131e8785e8d4eed9d1b5f0b72eb6585f988bc0da8ba084c8679361f7074a" Oct 30 06:06:27.137215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.137126 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:27.137215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.137195 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:27.410204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.410124 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr"] Oct 30 06:06:27.411806 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.411771 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c959a373-cca2-4f23-bac1-05b1c4eba284" containerName="revoke-keys" Oct 30 06:06:27.411806 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.411795 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c959a373-cca2-4f23-bac1-05b1c4eba284" containerName="revoke-keys" Oct 30 06:06:27.412018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.411952 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c959a373-cca2-4f23-bac1-05b1c4eba284" containerName="revoke-keys" Oct 30 06:06:27.415396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.415370 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.432281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.432258 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-1b4b7a-openshift-default-dockercfg-w77rm\"" Oct 30 06:06:27.451359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.451337 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr"] Oct 30 06:06:27.484618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484519 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.484821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484632 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.484821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484693 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.484821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484722 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.484821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484743 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.484821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484808 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.485114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484829 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxx2h\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.485114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484864 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.485114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.484918 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586298 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586339 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586387 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586411 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586432 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586500 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586520 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxx2h\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586585 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586639 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.586851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586785 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.587003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.586986 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.587082 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.587061 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.587334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.587310 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.587440 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.587421 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.589279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.589244 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.589405 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.589386 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.613779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.613761 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.613882 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.613840 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxx2h\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h\") pod \"e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:27.730880 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:27.730803 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:28.092142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.092113 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr"] Oct 30 06:06:28.093781 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:06:28.093750 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0f65c9b_058b_4e1b_9dff_54d86121798c.slice/crio-cc6fad37f63ce223ab04781a1488aa50d7d579f20eb833d32143cf05cedf616a WatchSource:0}: Error finding container cc6fad37f63ce223ab04781a1488aa50d7d579f20eb833d32143cf05cedf616a: Status 404 returned error can't find the container with id cc6fad37f63ce223ab04781a1488aa50d7d579f20eb833d32143cf05cedf616a Oct 30 06:06:28.097798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.097759 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:28.097977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.097950 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:28.098054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.098009 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:06:28.137271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.137246 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:28.137676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.137297 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:28.808651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.808614 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" event={"ID":"f0f65c9b-058b-4e1b-9dff-54d86121798c","Type":"ContainerStarted","Data":"3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113"} Oct 30 06:06:28.808859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:28.808673 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" event={"ID":"f0f65c9b-058b-4e1b-9dff-54d86121798c","Type":"ContainerStarted","Data":"cc6fad37f63ce223ab04781a1488aa50d7d579f20eb833d32143cf05cedf616a"} Oct 30 06:06:29.136591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:29.136496 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:29.136736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:29.136591 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:29.731020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:29.730990 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:29.732955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:29.732932 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:06:29.733082 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:29.732976 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 06:06:30.137279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:30.137244 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" start-of-body= Oct 30 06:06:30.137503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:30.137303 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.215:15021/healthz/ready\": dial tcp 10.134.0.215:15021: connect: connection refused" Oct 30 06:06:30.731592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:30.731556 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:06:30.731988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:30.731612 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 06:06:31.141858 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:31.141827 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:31.142295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:31.142279 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:31.143113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:31.143096 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:06:31.214011 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:31.213964 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podStartSLOduration=4.213949047 podStartE2EDuration="4.213949047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:28.87888252 +0000 UTC m=+2095.454569576" watchObservedRunningTime="2026-10-30 06:06:31.213949047 +0000 UTC m=+2097.789636101" Oct 30 06:06:31.731493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:31.731452 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:06:31.731734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:31.731498 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 06:06:32.729922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:32.729887 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9"] Oct 30 06:06:32.731781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:32.731749 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:06:32.732120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:32.731797 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 06:06:33.731410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:33.731381 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:06:33.731593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:33.731427 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 06:06:34.731430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:34.731387 5565 patch_prober.go:29] interesting pod/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body= Oct 30 06:06:34.731854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:34.731464 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" Oct 30 06:06:35.772825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:35.772789 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:35.774631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:35.774453 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:35.776350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:35.776317 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:06:37.035515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:37.035482 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:06:47.538448 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:06:47.538406 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:06:47.787762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:47.787687 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" podUID="5fbbec0e-d71e-46fb-94bb-90eca336c523" containerName="main" containerID="cri-o://df72590602472490e1377072c5f733cb9ac0efefed24d6cc920bdbcb552eb08f" gracePeriod=45 Oct 30 06:06:48.029132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.029098 5565 generic.go:356] "Generic (PLEG): container finished" podID="5fbbec0e-d71e-46fb-94bb-90eca336c523" containerID="df72590602472490e1377072c5f733cb9ac0efefed24d6cc920bdbcb552eb08f" exitCode=0 Oct 30 06:06:48.029337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.029171 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" event={"ID":"5fbbec0e-d71e-46fb-94bb-90eca336c523","Type":"ContainerDied","Data":"df72590602472490e1377072c5f733cb9ac0efefed24d6cc920bdbcb552eb08f"} Oct 30 06:06:48.112651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.112627 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:06:48.153370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153339 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache\") pod \"5fbbec0e-d71e-46fb-94bb-90eca336c523\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " Oct 30 06:06:48.153495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153439 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv\" (UniqueName: \"kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv\") pod \"5fbbec0e-d71e-46fb-94bb-90eca336c523\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " Oct 30 06:06:48.153616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153598 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs\") pod \"5fbbec0e-d71e-46fb-94bb-90eca336c523\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " Oct 30 06:06:48.153661 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153615 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache" pod "5fbbec0e-d71e-46fb-94bb-90eca336c523" (UID: "5fbbec0e-d71e-46fb-94bb-90eca336c523"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:48.153852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153820 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir\") pod \"5fbbec0e-d71e-46fb-94bb-90eca336c523\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " Oct 30 06:06:48.154024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153900 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm\") pod \"5fbbec0e-d71e-46fb-94bb-90eca336c523\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " Oct 30 06:06:48.154024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.153934 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home\") pod \"5fbbec0e-d71e-46fb-94bb-90eca336c523\" (UID: \"5fbbec0e-d71e-46fb-94bb-90eca336c523\") " Oct 30 06:06:48.154140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.154050 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir" pod "5fbbec0e-d71e-46fb-94bb-90eca336c523" (UID: "5fbbec0e-d71e-46fb-94bb-90eca336c523"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:48.154326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.154271 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home" pod "5fbbec0e-d71e-46fb-94bb-90eca336c523" (UID: "5fbbec0e-d71e-46fb-94bb-90eca336c523"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:48.154898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.154875 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:48.154982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.154906 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:48.154982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.154946 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:48.156146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.156115 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm" pod "5fbbec0e-d71e-46fb-94bb-90eca336c523" (UID: "5fbbec0e-d71e-46fb-94bb-90eca336c523"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:06:48.156258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.156144 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs" pod "5fbbec0e-d71e-46fb-94bb-90eca336c523" (UID: "5fbbec0e-d71e-46fb-94bb-90eca336c523"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:48.156702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.156675 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv" pod "5fbbec0e-d71e-46fb-94bb-90eca336c523" (UID: "5fbbec0e-d71e-46fb-94bb-90eca336c523"). InnerVolumeSpecName "kube-api-access-tffxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:48.257128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.257096 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5fbbec0e-d71e-46fb-94bb-90eca336c523-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:48.257128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.257129 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5fbbec0e-d71e-46fb-94bb-90eca336c523-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:48.257308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:48.257145 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tffxv\" (UniqueName: \"kubernetes.io/projected/5fbbec0e-d71e-46fb-94bb-90eca336c523-kube-api-access-tffxv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:49.038759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:49.038726 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" Oct 30 06:06:49.039200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:49.038731 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9" event={"ID":"5fbbec0e-d71e-46fb-94bb-90eca336c523","Type":"ContainerDied","Data":"8d4ae2cf95da48f98348da2d1c28372ec3b4459860e3a4b666c090aeda79d70d"} Oct 30 06:06:49.039200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:49.038863 5565 scope.go:122] "RemoveContainer" containerID="df72590602472490e1377072c5f733cb9ac0efefed24d6cc920bdbcb552eb08f" Oct 30 06:06:49.098424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:49.098393 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9"] Oct 30 06:06:49.111903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:49.111875 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-086c60/test-model-086c60-kserve-689fd654fc-8pcb9"] Oct 30 06:06:49.571359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:49.571323 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fbbec0e-d71e-46fb-94bb-90eca336c523" path="/var/lib/kubelet/pods/5fbbec0e-d71e-46fb-94bb-90eca336c523/volumes" Oct 30 06:06:52.668013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.667982 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8"] Oct 30 06:06:52.668821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.668804 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5fbbec0e-d71e-46fb-94bb-90eca336c523" containerName="main" Oct 30 06:06:52.668867 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.668821 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fbbec0e-d71e-46fb-94bb-90eca336c523" containerName="main" Oct 30 06:06:52.668961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.668952 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fbbec0e-d71e-46fb-94bb-90eca336c523" containerName="main" Oct 30 06:06:52.685019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.684988 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.694672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.694335 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8"] Oct 30 06:06:52.701925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.701898 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:06:52.702136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.702116 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:06:52.836958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.836924 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6b6c6\" (UniqueName: \"kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.837121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.837065 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.837173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.837120 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.938576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.938482 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.938743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.938721 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6b6c6\" (UniqueName: \"kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.938853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.938837 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.939456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.939431 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.940917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.940894 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:52.964024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:52.964003 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6b6c6\" (UniqueName: \"kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6\") pod \"maas-api-revoke-sub-01269541-tjpd8\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:53.010318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:53.010295 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:53.322028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:53.321999 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8"] Oct 30 06:06:53.322358 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:06:53.322332 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e477b94_9bb7_4437_bc29_ebea4fbdd33b.slice/crio-58673166be346bdd99bfdac4870e3cec16f3b052c63bb2d21fa3a856f9f4c385 WatchSource:0}: Error finding container 58673166be346bdd99bfdac4870e3cec16f3b052c63bb2d21fa3a856f9f4c385: Status 404 returned error can't find the container with id 58673166be346bdd99bfdac4870e3cec16f3b052c63bb2d21fa3a856f9f4c385 Oct 30 06:06:54.085055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:54.085018 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" event={"ID":"0e477b94-9bb7-4437-bc29-ebea4fbdd33b","Type":"ContainerStarted","Data":"afa8c26a29639effff852ca3b26563ec7c6b71ac0bd8c3ebb9420e711cd632eb"} Oct 30 06:06:54.085453 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:54.085077 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" event={"ID":"0e477b94-9bb7-4437-bc29-ebea4fbdd33b","Type":"ContainerStarted","Data":"58673166be346bdd99bfdac4870e3cec16f3b052c63bb2d21fa3a856f9f4c385"} Oct 30 06:06:54.153325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:54.147830 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" podStartSLOduration=2.1478086530000002 podStartE2EDuration="2.147808653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:06:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:06:54.129214028 +0000 UTC m=+2120.704901138" watchObservedRunningTime="2026-10-30 06:06:54.147808653 +0000 UTC m=+2120.723495707" Oct 30 06:06:55.094402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:55.094368 5565 generic.go:356] "Generic (PLEG): container finished" podID="0e477b94-9bb7-4437-bc29-ebea4fbdd33b" containerID="afa8c26a29639effff852ca3b26563ec7c6b71ac0bd8c3ebb9420e711cd632eb" exitCode=0 Oct 30 06:06:55.094840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:55.094450 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" event={"ID":"0e477b94-9bb7-4437-bc29-ebea4fbdd33b","Type":"ContainerDied","Data":"afa8c26a29639effff852ca3b26563ec7c6b71ac0bd8c3ebb9420e711cd632eb"} Oct 30 06:06:55.461561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:55.461465 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt"] Oct 30 06:06:56.336091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.336067 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:06:56.434933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.434854 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca\") pod \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " Oct 30 06:06:56.435307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.435275 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca\") pod \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " Oct 30 06:06:56.435433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.435377 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6\" (UniqueName: \"kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6\") pod \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\" (UID: \"0e477b94-9bb7-4437-bc29-ebea4fbdd33b\") " Oct 30 06:06:56.435819 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.435790 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca" pod "0e477b94-9bb7-4437-bc29-ebea4fbdd33b" (UID: "0e477b94-9bb7-4437-bc29-ebea4fbdd33b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:06:56.436092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.436069 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:56.438132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.438103 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6" pod "0e477b94-9bb7-4437-bc29-ebea4fbdd33b" (UID: "0e477b94-9bb7-4437-bc29-ebea4fbdd33b"). InnerVolumeSpecName "kube-api-access-6b6c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:06:56.438365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.438340 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca" pod "0e477b94-9bb7-4437-bc29-ebea4fbdd33b" (UID: "0e477b94-9bb7-4437-bc29-ebea4fbdd33b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:06:56.537230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.537203 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:56.537230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:56.537227 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6b6c6\" (UniqueName: \"kubernetes.io/projected/0e477b94-9bb7-4437-bc29-ebea4fbdd33b-kube-api-access-6b6c6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:06:57.155517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:57.155483 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" event={"ID":"0e477b94-9bb7-4437-bc29-ebea4fbdd33b","Type":"ContainerDied","Data":"58673166be346bdd99bfdac4870e3cec16f3b052c63bb2d21fa3a856f9f4c385"} Oct 30 06:06:57.155716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:57.155561 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="58673166be346bdd99bfdac4870e3cec16f3b052c63bb2d21fa3a856f9f4c385" Oct 30 06:06:57.155716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:06:57.155494 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8" Oct 30 06:07:03.828077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.828042 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c"] Oct 30 06:07:03.830042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.830013 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0e477b94-9bb7-4437-bc29-ebea4fbdd33b" containerName="revoke-keys" Oct 30 06:07:03.830208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.830196 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e477b94-9bb7-4437-bc29-ebea4fbdd33b" containerName="revoke-keys" Oct 30 06:07:03.830481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.830465 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e477b94-9bb7-4437-bc29-ebea4fbdd33b" containerName="revoke-keys" Oct 30 06:07:03.835334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.835309 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:03.850226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.850203 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c"] Oct 30 06:07:03.857229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.857205 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:07:03.858209 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:03.857344 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:07:04.003264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.003226 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.003438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.003380 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v559l\" (UniqueName: \"kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.003604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.003585 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.104953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.104866 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.105120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.104991 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.105120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.105108 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v559l\" (UniqueName: \"kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.105574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.105549 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.108086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.108065 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.130568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.130517 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v559l\" (UniqueName: \"kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l\") pod \"maas-api-revoke-sub-f51265a6-tzn2c\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.153635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.153613 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:04.211587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.207607 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg"] Oct 30 06:07:04.216403 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.216352 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-765de9-g46hg"] Oct 30 06:07:04.474791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:04.474751 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c"] Oct 30 06:07:05.234046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:05.233922 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" event={"ID":"863e15bc-6719-4783-970d-a757bddbe9ec","Type":"ContainerStarted","Data":"48c41e483e344685343648cd456300d987a8bf733924a5ae702ad7299e80fe17"} Oct 30 06:07:05.234556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:05.234055 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" event={"ID":"863e15bc-6719-4783-970d-a757bddbe9ec","Type":"ContainerStarted","Data":"471c896600f6b8548a455c2a462b9192d93cf0ec08e8c1f66683ccd39ab4f0f2"} Oct 30 06:07:05.282485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:05.282438 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" podStartSLOduration=2.282423004 podStartE2EDuration="2.282423004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:07:05.278396643 +0000 UTC m=+2131.854083697" watchObservedRunningTime="2026-10-30 06:07:05.282423004 +0000 UTC m=+2131.858110060" Oct 30 06:07:05.570727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:05.570693 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7" path="/var/lib/kubelet/pods/246fbf1b-0fc1-4f2b-bfb2-f7d2dd0390d7/volumes" Oct 30 06:07:06.247492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:06.247450 5565 generic.go:356] "Generic (PLEG): container finished" podID="863e15bc-6719-4783-970d-a757bddbe9ec" containerID="48c41e483e344685343648cd456300d987a8bf733924a5ae702ad7299e80fe17" exitCode=0 Oct 30 06:07:06.247985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:06.247552 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" event={"ID":"863e15bc-6719-4783-970d-a757bddbe9ec","Type":"ContainerDied","Data":"48c41e483e344685343648cd456300d987a8bf733924a5ae702ad7299e80fe17"} Oct 30 06:07:07.433235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.433210 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:07.502239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.502210 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca\") pod \"863e15bc-6719-4783-970d-a757bddbe9ec\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " Oct 30 06:07:07.502419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.502383 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca\") pod \"863e15bc-6719-4783-970d-a757bddbe9ec\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " Oct 30 06:07:07.502632 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.502610 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l\" (UniqueName: \"kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l\") pod \"863e15bc-6719-4783-970d-a757bddbe9ec\" (UID: \"863e15bc-6719-4783-970d-a757bddbe9ec\") " Oct 30 06:07:07.502841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.502813 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca" pod "863e15bc-6719-4783-970d-a757bddbe9ec" (UID: "863e15bc-6719-4783-970d-a757bddbe9ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:07.503655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.503631 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:07.504822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.504791 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l" pod "863e15bc-6719-4783-970d-a757bddbe9ec" (UID: "863e15bc-6719-4783-970d-a757bddbe9ec"). InnerVolumeSpecName "kube-api-access-v559l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:07.504930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.504860 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca" pod "863e15bc-6719-4783-970d-a757bddbe9ec" (UID: "863e15bc-6719-4783-970d-a757bddbe9ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:07.605641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.605603 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/863e15bc-6719-4783-970d-a757bddbe9ec-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:07.605641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:07.605641 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v559l\" (UniqueName: \"kubernetes.io/projected/863e15bc-6719-4783-970d-a757bddbe9ec-kube-api-access-v559l\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:08.269024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:08.268966 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" event={"ID":"863e15bc-6719-4783-970d-a757bddbe9ec","Type":"ContainerDied","Data":"471c896600f6b8548a455c2a462b9192d93cf0ec08e8c1f66683ccd39ab4f0f2"} Oct 30 06:07:08.269024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:08.269043 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="471c896600f6b8548a455c2a462b9192d93cf0ec08e8c1f66683ccd39ab4f0f2" Oct 30 06:07:08.269443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:08.269120 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c" Oct 30 06:07:10.483715 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.483668 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" podUID="e30c4190-1c92-415a-9c32-1a48773963d4" containerName="main" containerID="cri-o://d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e" gracePeriod=45 Oct 30 06:07:10.808643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.808615 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:07:10.871154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871119 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache\") pod \"e30c4190-1c92-415a-9c32-1a48773963d4\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " Oct 30 06:07:10.871337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871215 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm\") pod \"e30c4190-1c92-415a-9c32-1a48773963d4\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " Oct 30 06:07:10.871337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871258 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8\" (UniqueName: \"kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8\") pod \"e30c4190-1c92-415a-9c32-1a48773963d4\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " Oct 30 06:07:10.871472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871408 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache" pod "e30c4190-1c92-415a-9c32-1a48773963d4" (UID: "e30c4190-1c92-415a-9c32-1a48773963d4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:10.871472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871435 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home\") pod \"e30c4190-1c92-415a-9c32-1a48773963d4\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " Oct 30 06:07:10.871625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871491 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs\" (UniqueName: \"kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs\") pod \"e30c4190-1c92-415a-9c32-1a48773963d4\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " Oct 30 06:07:10.871687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871626 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir\") pod \"e30c4190-1c92-415a-9c32-1a48773963d4\" (UID: \"e30c4190-1c92-415a-9c32-1a48773963d4\") " Oct 30 06:07:10.871781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.871740 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home" pod "e30c4190-1c92-415a-9c32-1a48773963d4" (UID: "e30c4190-1c92-415a-9c32-1a48773963d4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:10.872066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.872040 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir" pod "e30c4190-1c92-415a-9c32-1a48773963d4" (UID: "e30c4190-1c92-415a-9c32-1a48773963d4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:10.872599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.872576 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:10.872714 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.872603 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:10.872714 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.872616 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:10.873911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.873886 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs" pod "e30c4190-1c92-415a-9c32-1a48773963d4" (UID: "e30c4190-1c92-415a-9c32-1a48773963d4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:07:10.874006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.873887 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm" pod "e30c4190-1c92-415a-9c32-1a48773963d4" (UID: "e30c4190-1c92-415a-9c32-1a48773963d4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:10.878702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.878683 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8" pod "e30c4190-1c92-415a-9c32-1a48773963d4" (UID: "e30c4190-1c92-415a-9c32-1a48773963d4"). InnerVolumeSpecName "kube-api-access-p95q8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:10.979555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.977230 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e30c4190-1c92-415a-9c32-1a48773963d4-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:10.979555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.977268 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p95q8\" (UniqueName: \"kubernetes.io/projected/e30c4190-1c92-415a-9c32-1a48773963d4-kube-api-access-p95q8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:10.979555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:10.978339 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e30c4190-1c92-415a-9c32-1a48773963d4-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:11.227209 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.227123 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp"] Oct 30 06:07:11.227388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.227318 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" containerID="cri-o://e943d8064d3934fc0cc779d5993604cbbebd3f233772dbe09b13db348c2c860d" gracePeriod=30 Oct 30 06:07:11.419549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.418900 5565 generic.go:356] "Generic (PLEG): container finished" podID="e30c4190-1c92-415a-9c32-1a48773963d4" containerID="d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e" exitCode=0 Oct 30 06:07:11.419549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.419152 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" event={"ID":"e30c4190-1c92-415a-9c32-1a48773963d4","Type":"ContainerDied","Data":"d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e"} Oct 30 06:07:11.419549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.419188 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" event={"ID":"e30c4190-1c92-415a-9c32-1a48773963d4","Type":"ContainerDied","Data":"583fae335c9f2bc7fc1c887996250f57e3389acb42408d12c16d993872b8500b"} Oct 30 06:07:11.419549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.419209 5565 scope.go:122] "RemoveContainer" containerID="d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e" Oct 30 06:07:11.419549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.419368 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt" Oct 30 06:07:11.611773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.607092 5565 scope.go:122] "RemoveContainer" containerID="d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e" Oct 30 06:07:11.611773 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:07:11.609663 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e\": container with ID starting with d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e not found: ID does not exist" containerID="d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e" Oct 30 06:07:11.611773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.609705 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e"} err="failed to get container status \"d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e\": rpc error: code = NotFound desc = could not find container \"d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e\": container with ID starting with d250d78597efec4b7a9ed3d6d52d9498514eea67124c17199bea015898511f4e not found: ID does not exist" Oct 30 06:07:11.654820 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.654789 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt"] Oct 30 06:07:11.658599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:11.658574 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-8de592/test-model-8de592-kserve-d78bd6ccc-j8fjt"] Oct 30 06:07:13.570193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:13.570155 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e30c4190-1c92-415a-9c32-1a48773963d4" path="/var/lib/kubelet/pods/e30c4190-1c92-415a-9c32-1a48773963d4/volumes" Oct 30 06:07:16.466756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.466714 5565 generic.go:356] "Generic (PLEG): container finished" podID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerID="e943d8064d3934fc0cc779d5993604cbbebd3f233772dbe09b13db348c2c860d" exitCode=0 Oct 30 06:07:16.467202 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.466768 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" event={"ID":"48b14bc0-c5dd-4f8c-b29a-62da80c63a93","Type":"ContainerDied","Data":"e943d8064d3934fc0cc779d5993604cbbebd3f233772dbe09b13db348c2c860d"} Oct 30 06:07:16.544670 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.544640 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:07:16.596170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596135 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596221 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596327 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596376 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596443 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596511 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596559 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596652 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.596762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596660 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.596917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596880 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket\") pod \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\" (UID: \"48b14bc0-c5dd-4f8c-b29a-62da80c63a93\") " Oct 30 06:07:16.597175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596888 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.597262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.596908 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.597262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.597244 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:07:16.597347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.597313 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.598107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598063 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.598107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598095 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.598293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598112 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.598293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598125 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.598293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598140 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.598724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598697 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "kube-api-access-7bp76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:16.598942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.598913 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:07:16.599031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.599009 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:07:16.599450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.599410 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token" pod "48b14bc0-c5dd-4f8c-b29a-62da80c63a93" (UID: "48b14bc0-c5dd-4f8c-b29a-62da80c63a93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:07:16.700245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.700203 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bp76\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-kube-api-access-7bp76\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.700245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.700240 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.700245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.700252 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:16.700500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:16.700266 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48b14bc0-c5dd-4f8c-b29a-62da80c63a93-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:07:17.476854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:17.476823 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" Oct 30 06:07:17.477288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:17.476857 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp" event={"ID":"48b14bc0-c5dd-4f8c-b29a-62da80c63a93","Type":"ContainerDied","Data":"659237ea16fc88ddb56f71595468b429cb4a015cbfe4e6df6ee30a4a6f7a3497"} Oct 30 06:07:17.477288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:17.476969 5565 scope.go:122] "RemoveContainer" containerID="e943d8064d3934fc0cc779d5993604cbbebd3f233772dbe09b13db348c2c860d" Oct 30 06:07:17.534360 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:07:17.534329 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:07:17.552410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:17.552381 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp"] Oct 30 06:07:17.570722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:17.570687 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-765de9-openshift-default-86b6484b6f-xvgjp"] Oct 30 06:07:19.569727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:19.569695 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" path="/var/lib/kubelet/pods/48b14bc0-c5dd-4f8c-b29a-62da80c63a93/volumes" Oct 30 06:07:38.025124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:38.025092 5565 scope.go:122] "RemoveContainer" containerID="84362b4297f954e13b94994fb2547b0534a2c64ff556ec7e0e787171b2552fe8" Oct 30 06:07:38.035782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:38.035761 5565 scope.go:122] "RemoveContainer" containerID="b061a779c544ba32c6ded98091e3b0d612c55902f26fad261fe92dc483c1a18a" Oct 30 06:07:47.533347 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:07:47.533256 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:07:58.522809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.522774 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq"] Oct 30 06:07:58.523621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523604 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e30c4190-1c92-415a-9c32-1a48773963d4" containerName="main" Oct 30 06:07:58.523679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523621 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e30c4190-1c92-415a-9c32-1a48773963d4" containerName="main" Oct 30 06:07:58.523679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523641 5565 cpu_manager.go:414] "containerMap: removing container" podUID="863e15bc-6719-4783-970d-a757bddbe9ec" containerName="revoke-keys" Oct 30 06:07:58.523679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523648 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="863e15bc-6719-4783-970d-a757bddbe9ec" containerName="revoke-keys" Oct 30 06:07:58.523785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523685 5565 cpu_manager.go:414] "containerMap: removing container" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" Oct 30 06:07:58.523785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523690 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" Oct 30 06:07:58.523850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523810 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e30c4190-1c92-415a-9c32-1a48773963d4" containerName="main" Oct 30 06:07:58.523850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523819 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="863e15bc-6719-4783-970d-a757bddbe9ec" containerName="revoke-keys" Oct 30 06:07:58.523850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.523832 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="48b14bc0-c5dd-4f8c-b29a-62da80c63a93" containerName="istio-proxy" Oct 30 06:07:58.529301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.529280 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.550576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.550548 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq"] Oct 30 06:07:58.555542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.555511 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:07:58.555686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.555511 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:07:58.638196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.638150 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.638587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.638473 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.638587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.638578 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsnrq\" (UniqueName: \"kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.751557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.741460 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.759770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.759734 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsnrq\" (UniqueName: \"kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.759954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.759816 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.760620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.758941 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.773651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.762912 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.787983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.787955 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsnrq\" (UniqueName: \"kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq\") pod \"maas-api-revoke-sub-518c4c4e-g6czq\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:58.848815 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:58.848783 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:07:59.157120 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:07:59.157088 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac4d83ca_9be4_42e7_95b6_2e8110ec5943.slice/crio-0cef0dfb402c7c5220a5f3ecb7e37693ce02a1210e1b26f75b52832e11920546 WatchSource:0}: Error finding container 0cef0dfb402c7c5220a5f3ecb7e37693ce02a1210e1b26f75b52832e11920546: Status 404 returned error can't find the container with id 0cef0dfb402c7c5220a5f3ecb7e37693ce02a1210e1b26f75b52832e11920546 Oct 30 06:07:59.157892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:07:59.157858 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq"] Oct 30 06:08:00.028594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:00.028559 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" event={"ID":"ac4d83ca-9be4-42e7-95b6-2e8110ec5943","Type":"ContainerStarted","Data":"20aa1093cc7b44402782b3ea93dc633f63139d8844710cf7f5f633ee465ad166"} Oct 30 06:08:00.029005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:00.028617 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" event={"ID":"ac4d83ca-9be4-42e7-95b6-2e8110ec5943","Type":"ContainerStarted","Data":"0cef0dfb402c7c5220a5f3ecb7e37693ce02a1210e1b26f75b52832e11920546"} Oct 30 06:08:00.077607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:00.077558 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" podStartSLOduration=2.077543266 podStartE2EDuration="2.077543266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:07:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:00.072917225 +0000 UTC m=+2186.648604280" watchObservedRunningTime="2026-10-30 06:08:00.077543266 +0000 UTC m=+2186.653230319" Oct 30 06:08:01.038249 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:01.038214 5565 generic.go:356] "Generic (PLEG): container finished" podID="ac4d83ca-9be4-42e7-95b6-2e8110ec5943" containerID="20aa1093cc7b44402782b3ea93dc633f63139d8844710cf7f5f633ee465ad166" exitCode=0 Oct 30 06:08:01.038704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:01.038293 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" event={"ID":"ac4d83ca-9be4-42e7-95b6-2e8110ec5943","Type":"ContainerDied","Data":"20aa1093cc7b44402782b3ea93dc633f63139d8844710cf7f5f633ee465ad166"} Oct 30 06:08:02.115425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.115392 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn"] Oct 30 06:08:02.115906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.115649 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" containerID="cri-o://b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d" gracePeriod=30 Oct 30 06:08:02.132598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.130718 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p"] Oct 30 06:08:02.132598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.130951 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" containerID="cri-o://2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238" gracePeriod=30 Oct 30 06:08:02.136870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.136833 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt"] Oct 30 06:08:02.139917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.138464 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" containerID="cri-o://42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2" gracePeriod=30 Oct 30 06:08:02.532819 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.532791 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:08:02.584962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.584931 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca\") pod \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " Oct 30 06:08:02.585113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.585000 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca\") pod \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " Oct 30 06:08:02.585202 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.585178 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq\" (UniqueName: \"kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq\") pod \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\" (UID: \"ac4d83ca-9be4-42e7-95b6-2e8110ec5943\") " Oct 30 06:08:02.585346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.585324 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca" pod "ac4d83ca-9be4-42e7-95b6-2e8110ec5943" (UID: "ac4d83ca-9be4-42e7-95b6-2e8110ec5943"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:02.586515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.586487 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.587700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.587676 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca" pod "ac4d83ca-9be4-42e7-95b6-2e8110ec5943" (UID: "ac4d83ca-9be4-42e7-95b6-2e8110ec5943"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:02.587839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.587821 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq" pod "ac4d83ca-9be4-42e7-95b6-2e8110ec5943" (UID: "ac4d83ca-9be4-42e7-95b6-2e8110ec5943"). InnerVolumeSpecName "kube-api-access-lsnrq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:02.688355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.688313 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:02.688355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:02.688352 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsnrq\" (UniqueName: \"kubernetes.io/projected/ac4d83ca-9be4-42e7-95b6-2e8110ec5943-kube-api-access-lsnrq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:03.055091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:03.055066 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" Oct 30 06:08:03.055245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:03.055101 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq" event={"ID":"ac4d83ca-9be4-42e7-95b6-2e8110ec5943","Type":"ContainerDied","Data":"0cef0dfb402c7c5220a5f3ecb7e37693ce02a1210e1b26f75b52832e11920546"} Oct 30 06:08:03.055245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:03.055148 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cef0dfb402c7c5220a5f3ecb7e37693ce02a1210e1b26f75b52832e11920546" Oct 30 06:08:06.116607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:06.116572 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g"] Oct 30 06:08:06.117264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:06.116810 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" containerID="cri-o://b1f68ac0f15ff0208038638b270c7c7bcd4e628e99c4c1efc1067842ed7c5437" gracePeriod=30 Oct 30 06:08:06.666577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:06.666543 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:08:07.538695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.538654 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:08:07.589759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.589717 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.589959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.589872 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.589959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.589915 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.590312 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.590263 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.590414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.590366 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.590414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.590399 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.590639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.590487 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.590797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.590769 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.592020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.591986 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:07.592562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.592515 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.592777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.592755 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.592848 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.591787 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.592912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.592891 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs\") pod \"09e41011-96e4-4203-9cf7-83c8a991aff1\" (UID: \"09e41011-96e4-4203-9cf7-83c8a991aff1\") " Oct 30 06:08:07.594285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.594251 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.594419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.594382 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.594483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.594426 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09e41011-96e4-4203-9cf7-83c8a991aff1-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.594483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.594441 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.594483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.594455 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.594702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.594668 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.597576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.597118 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "kube-api-access-pm9sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.599478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.599396 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.625815 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.624851 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo" pod "09e41011-96e4-4203-9cf7-83c8a991aff1" (UID: "09e41011-96e4-4203-9cf7-83c8a991aff1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:07.641776 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.641589 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:08:07.646923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.646820 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:08:07.695233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695198 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.695446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695256 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.695446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695285 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.695446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695325 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.695446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695370 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.695446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695424 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.695769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695486 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.695769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695545 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.695769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695648 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.695769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695687 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.695769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695750 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.696038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695819 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.696038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695862 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.696038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695958 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.696038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.695981 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.696369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.696095 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.696369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.696239 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.696369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.696304 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo\") pod \"b9e6953c-e2a4-4f78-9050-58b08787366b\" (UID: \"b9e6953c-e2a4-4f78-9050-58b08787366b\") " Oct 30 06:08:07.696369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.696346 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.696666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.696449 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk\") pod \"90d588f4-8fe6-417c-8e61-7b644f8190a3\" (UID: \"90d588f4-8fe6-417c-8e61-7b644f8190a3\") " Oct 30 06:08:07.696666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.696592 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:07.697268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697248 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697271 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697286 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm9sj\" (UniqueName: \"kubernetes.io/projected/09e41011-96e4-4203-9cf7-83c8a991aff1-kube-api-access-pm9sj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697299 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09e41011-96e4-4203-9cf7-83c8a991aff1-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697314 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09e41011-96e4-4203-9cf7-83c8a991aff1-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697329 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697343 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/90d588f4-8fe6-417c-8e61-7b644f8190a3-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697356 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.697941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.697921 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.698115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.698096 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.698608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.698579 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.698608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.698586 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.698810 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.698788 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.700208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.700132 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.700594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.700564 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "kube-api-access-h2fhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.700948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.700895 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:07.701921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.701896 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "kube-api-access-hgkbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.702715 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.702690 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.702950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.702927 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.703033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.702993 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo" pod "90d588f4-8fe6-417c-8e61-7b644f8190a3" (UID: "90d588f4-8fe6-417c-8e61-7b644f8190a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:07.703033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.702998 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:07.704575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.704509 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:07.706737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.706706 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy" pod "b9e6953c-e2a4-4f78-9050-58b08787366b" (UID: "b9e6953c-e2a4-4f78-9050-58b08787366b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:07.799485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799436 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799485 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799500 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799516 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9e6953c-e2a4-4f78-9050-58b08787366b-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799548 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799561 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799577 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799591 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgkbt\" (UniqueName: \"kubernetes.io/projected/b9e6953c-e2a4-4f78-9050-58b08787366b-kube-api-access-hgkbt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799605 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799619 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799634 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799647 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9e6953c-e2a4-4f78-9050-58b08787366b-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799660 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9e6953c-e2a4-4f78-9050-58b08787366b-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799672 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/90d588f4-8fe6-417c-8e61-7b644f8190a3-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:07.799744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:07.799688 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2fhk\" (UniqueName: \"kubernetes.io/projected/90d588f4-8fe6-417c-8e61-7b644f8190a3-kube-api-access-h2fhk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:08.174426 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.174373 5565 generic.go:356] "Generic (PLEG): container finished" podID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerID="b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d" exitCode=0 Oct 30 06:08:08.174852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.174812 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" event={"ID":"09e41011-96e4-4203-9cf7-83c8a991aff1","Type":"ContainerDied","Data":"b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d"} Oct 30 06:08:08.175060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.175041 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" event={"ID":"09e41011-96e4-4203-9cf7-83c8a991aff1","Type":"ContainerDied","Data":"7164932ea8326820c6267224f9f21d6c2fec1b4b4a9e8b93aa06ebce8a7f3e62"} Oct 30 06:08:08.175208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.175193 5565 scope.go:122] "RemoveContainer" containerID="b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d" Oct 30 06:08:08.175569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.175544 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn" Oct 30 06:08:08.225992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.225953 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" event={"ID":"90d588f4-8fe6-417c-8e61-7b644f8190a3","Type":"ContainerDied","Data":"42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2"} Oct 30 06:08:08.226355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.226325 5565 generic.go:356] "Generic (PLEG): container finished" podID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerID="42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2" exitCode=0 Oct 30 06:08:08.226703 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.226639 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" event={"ID":"90d588f4-8fe6-417c-8e61-7b644f8190a3","Type":"ContainerDied","Data":"89db2acbfa1980ec7ac8f5c3c93e043c2c334a4229d53d1027c5258f29cbd7c0"} Oct 30 06:08:08.226861 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.226842 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt" Oct 30 06:08:08.402598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.381659 5565 generic.go:356] "Generic (PLEG): container finished" podID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerID="2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238" exitCode=0 Oct 30 06:08:08.402598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.381740 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" event={"ID":"b9e6953c-e2a4-4f78-9050-58b08787366b","Type":"ContainerDied","Data":"2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238"} Oct 30 06:08:08.402598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.381804 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" event={"ID":"b9e6953c-e2a4-4f78-9050-58b08787366b","Type":"ContainerDied","Data":"f899e086f2d4ff07f078ac00627673f235a9528bb493989397493d75acbeeb1a"} Oct 30 06:08:08.402598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.381903 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p" Oct 30 06:08:08.465944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.459650 5565 scope.go:122] "RemoveContainer" containerID="b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d" Oct 30 06:08:08.470241 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:08:08.466267 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d\": container with ID starting with b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d not found: ID does not exist" containerID="b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d" Oct 30 06:08:08.470241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.466337 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d"} err="failed to get container status \"b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d\": rpc error: code = NotFound desc = could not find container \"b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d\": container with ID starting with b73400a4560cc6dff25c5ba37a9bd8c80b5c662d97ba203d49780e21d6b8629d not found: ID does not exist" Oct 30 06:08:08.470241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.466380 5565 scope.go:122] "RemoveContainer" containerID="42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2" Oct 30 06:08:08.529845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.528040 5565 scope.go:122] "RemoveContainer" containerID="42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2" Oct 30 06:08:08.529845 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:08:08.528521 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2\": container with ID starting with 42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2 not found: ID does not exist" containerID="42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2" Oct 30 06:08:08.529845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.529004 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2"} err="failed to get container status \"42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2\": rpc error: code = NotFound desc = could not find container \"42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2\": container with ID starting with 42db7990b09192bab73863ddd3cb25c6df0332c041ec0b8cbc9e87d08fbd59c2 not found: ID does not exist" Oct 30 06:08:08.529845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.529034 5565 scope.go:122] "RemoveContainer" containerID="2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238" Oct 30 06:08:08.653879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.653562 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p"] Oct 30 06:08:08.654713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.654686 5565 scope.go:122] "RemoveContainer" containerID="2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238" Oct 30 06:08:08.658549 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:08:08.656118 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238\": container with ID starting with 2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238 not found: ID does not exist" containerID="2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238" Oct 30 06:08:08.658549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.656161 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238"} err="failed to get container status \"2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238\": rpc error: code = NotFound desc = could not find container \"2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238\": container with ID starting with 2db2b0eb5686bc37d524a952342f909040cc78f06b15df059022fecd0f0f9238 not found: ID does not exist" Oct 30 06:08:08.691981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.681419 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-46f7p"] Oct 30 06:08:08.716717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.716250 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn"] Oct 30 06:08:08.728443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.728384 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-rtdrn"] Oct 30 06:08:08.766319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.766284 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt"] Oct 30 06:08:08.773105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:08.773067 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-086c60-openshift-default-59c6bd6697-444vt"] Oct 30 06:08:09.106565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:09.104614 5565 patch_prober.go:29] interesting pod/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:08:09.106565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:09.104678 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:08:09.573038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:09.572997 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" path="/var/lib/kubelet/pods/09e41011-96e4-4203-9cf7-83c8a991aff1/volumes" Oct 30 06:08:09.573688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:09.573666 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" path="/var/lib/kubelet/pods/90d588f4-8fe6-417c-8e61-7b644f8190a3/volumes" Oct 30 06:08:09.574211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:09.574191 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" path="/var/lib/kubelet/pods/b9e6953c-e2a4-4f78-9050-58b08787366b/volumes" Oct 30 06:08:11.418975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.418938 5565 generic.go:356] "Generic (PLEG): container finished" podID="27f64629-383b-4efd-95b2-84036908df7c" containerID="b1f68ac0f15ff0208038638b270c7c7bcd4e628e99c4c1efc1067842ed7c5437" exitCode=0 Oct 30 06:08:11.419474 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.419009 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" event={"ID":"27f64629-383b-4efd-95b2-84036908df7c","Type":"ContainerDied","Data":"b1f68ac0f15ff0208038638b270c7c7bcd4e628e99c4c1efc1067842ed7c5437"} Oct 30 06:08:11.419474 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.419064 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" event={"ID":"27f64629-383b-4efd-95b2-84036908df7c","Type":"ContainerDied","Data":"8cf5920ad3b3a4e44789fb6972fc18d49fb73d6d5307614e365d5926e8de47db"} Oct 30 06:08:11.419474 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.419085 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cf5920ad3b3a4e44789fb6972fc18d49fb73d6d5307614e365d5926e8de47db" Oct 30 06:08:11.431199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.431166 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 06:08:11.537277 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537239 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.537512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537387 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.537512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537422 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.537512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537479 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.537755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537572 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.537755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537648 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:11.537755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537672 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.537755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537744 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:11.538000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537804 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.538000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537912 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:11.538000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537950 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.538000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537979 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:11.538000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.537996 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token\") pod \"27f64629-383b-4efd-95b2-84036908df7c\" (UID: \"27f64629-383b-4efd-95b2-84036908df7c\") " Oct 30 06:08:11.538275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.538244 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:11.538910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.538879 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.539064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.538913 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.539064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.538930 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.539064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.538944 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.539064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.538958 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27f64629-383b-4efd-95b2-84036908df7c-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.541293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.541259 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:11.541676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.541541 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:11.541676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.541649 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "kube-api-access-4m99p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:11.541961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.541941 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token" pod "27f64629-383b-4efd-95b2-84036908df7c" (UID: "27f64629-383b-4efd-95b2-84036908df7c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:11.640988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.640886 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27f64629-383b-4efd-95b2-84036908df7c-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.640988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.640920 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m99p\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-kube-api-access-4m99p\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.640988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.640930 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27f64629-383b-4efd-95b2-84036908df7c-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:11.640988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:11.640938 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27f64629-383b-4efd-95b2-84036908df7c-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:12.427421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:12.427391 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g" Oct 30 06:08:12.503576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:12.503542 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g"] Oct 30 06:08:12.518619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:12.518590 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-cjq6g"] Oct 30 06:08:13.569167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:13.569132 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27f64629-383b-4efd-95b2-84036908df7c" path="/var/lib/kubelet/pods/27f64629-383b-4efd-95b2-84036908df7c/volumes" Oct 30 06:08:17.540188 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:08:17.540148 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:08:32.585109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585074 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns"] Oct 30 06:08:32.585933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585914 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ac4d83ca-9be4-42e7-95b6-2e8110ec5943" containerName="revoke-keys" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585934 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac4d83ca-9be4-42e7-95b6-2e8110ec5943" containerName="revoke-keys" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585952 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585958 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585973 5565 cpu_manager.go:414] "containerMap: removing container" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.585979 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586001 5565 cpu_manager.go:414] "containerMap: removing container" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586005 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" Oct 30 06:08:32.586013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586013 5565 cpu_manager.go:414] "containerMap: removing container" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" Oct 30 06:08:32.586267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586018 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" Oct 30 06:08:32.586267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586147 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac4d83ca-9be4-42e7-95b6-2e8110ec5943" containerName="revoke-keys" Oct 30 06:08:32.586267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586159 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9e6953c-e2a4-4f78-9050-58b08787366b" containerName="istio-proxy" Oct 30 06:08:32.586267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586165 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="27f64629-383b-4efd-95b2-84036908df7c" containerName="istio-proxy" Oct 30 06:08:32.586267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586176 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d588f4-8fe6-417c-8e61-7b644f8190a3" containerName="istio-proxy" Oct 30 06:08:32.586267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.586183 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="09e41011-96e4-4203-9cf7-83c8a991aff1" containerName="istio-proxy" Oct 30 06:08:32.589909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.589891 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.610718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.610691 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns"] Oct 30 06:08:32.620073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.620045 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:32.620212 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.620081 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:08:32.740302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.740265 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.740482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.740397 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54rrm\" (UniqueName: \"kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.740671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.740643 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.842961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.842871 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.843119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.843100 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.843232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.843216 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54rrm\" (UniqueName: \"kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.843824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.843799 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.845471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.845446 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.868261 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.868233 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54rrm\" (UniqueName: \"kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm\") pod \"maas-api-revoke-keys-e2e-inf-a-482468-84dns\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:32.905214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:32.905187 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:33.214998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:33.214968 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns"] Oct 30 06:08:33.216182 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:08:33.216146 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dc5d1f2_b788_4f6d_aca6_0f47da99b5ef.slice/crio-e19772abb1e006ab91f5c7a685cfe20b869d12d1dc263b43bc9baf9841b314d3 WatchSource:0}: Error finding container e19772abb1e006ab91f5c7a685cfe20b869d12d1dc263b43bc9baf9841b314d3: Status 404 returned error can't find the container with id e19772abb1e006ab91f5c7a685cfe20b869d12d1dc263b43bc9baf9841b314d3 Oct 30 06:08:33.611914 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:33.611882 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" event={"ID":"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef","Type":"ContainerStarted","Data":"4dfbe389b2816d47e37cc4e45a96e07636b86f8bb37c3f1ed516518c7e455d7c"} Oct 30 06:08:33.612301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:33.611933 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" event={"ID":"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef","Type":"ContainerStarted","Data":"e19772abb1e006ab91f5c7a685cfe20b869d12d1dc263b43bc9baf9841b314d3"} Oct 30 06:08:33.673966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:33.673911 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" podStartSLOduration=1.673897129 podStartE2EDuration="1.673897129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:08:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:08:33.65719913 +0000 UTC m=+2220.232886184" watchObservedRunningTime="2026-10-30 06:08:33.673897129 +0000 UTC m=+2220.249584221" Oct 30 06:08:34.622442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:34.622337 5565 generic.go:356] "Generic (PLEG): container finished" podID="8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" containerID="4dfbe389b2816d47e37cc4e45a96e07636b86f8bb37c3f1ed516518c7e455d7c" exitCode=0 Oct 30 06:08:34.622442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:34.622414 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" event={"ID":"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef","Type":"ContainerDied","Data":"4dfbe389b2816d47e37cc4e45a96e07636b86f8bb37c3f1ed516518c7e455d7c"} Oct 30 06:08:35.858632 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.858592 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:35.940726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.940635 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca\") pod \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " Oct 30 06:08:35.940726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.940678 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca\") pod \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " Oct 30 06:08:35.940984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.940735 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm\" (UniqueName: \"kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm\") pod \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\" (UID: \"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef\") " Oct 30 06:08:35.941145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.941112 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca" pod "8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" (UID: "8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:35.941624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.941598 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:35.943090 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.943043 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca" pod "8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" (UID: "8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:35.943185 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:35.943114 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm" pod "8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" (UID: "8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef"). InnerVolumeSpecName "kube-api-access-54rrm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:36.042764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:36.042723 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:36.042764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:36.042759 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54rrm\" (UniqueName: \"kubernetes.io/projected/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef-kube-api-access-54rrm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:36.642006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:36.641970 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" Oct 30 06:08:36.642006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:36.641996 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns" event={"ID":"8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef","Type":"ContainerDied","Data":"e19772abb1e006ab91f5c7a685cfe20b869d12d1dc263b43bc9baf9841b314d3"} Oct 30 06:08:36.642237 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:36.642045 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e19772abb1e006ab91f5c7a685cfe20b869d12d1dc263b43bc9baf9841b314d3" Oct 30 06:08:38.088196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:38.088153 5565 scope.go:122] "RemoveContainer" containerID="3ef486838c153efacc501af0d8dd80bd6672a900711be748e7600a884d02251a" Oct 30 06:08:38.102345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:38.102310 5565 scope.go:122] "RemoveContainer" containerID="5b8887094aecbe38e62297c3b415bffdf128c82c54f77b0affb9c7b44ed6853d" Oct 30 06:08:38.116769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:38.116746 5565 scope.go:122] "RemoveContainer" containerID="b1f68ac0f15ff0208038638b270c7c7bcd4e628e99c4c1efc1067842ed7c5437" Oct 30 06:08:42.211787 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:42.211753 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv"] Oct 30 06:08:42.212280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:42.212012 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" containerID="cri-o://e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d" gracePeriod=30 Oct 30 06:08:44.906604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.906564 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57"] Oct 30 06:08:44.910624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.908002 5565 cpu_manager.go:414] "containerMap: removing container" podUID="8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" containerName="revoke-keys" Oct 30 06:08:44.910624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.908030 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" containerName="revoke-keys" Oct 30 06:08:44.910624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.908242 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" containerName="revoke-keys" Oct 30 06:08:44.915377 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.915342 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:44.928041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.927701 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57"] Oct 30 06:08:44.935263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.935234 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:08:44.935568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:44.935491 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:08:45.008623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.008590 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.008623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.008627 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m72wh\" (UniqueName: \"kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.008843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.008807 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.110667 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.110624 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.110855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.110674 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m72wh\" (UniqueName: \"kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.110855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.110778 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.111420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.111393 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.113120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.113097 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.135962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.135940 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m72wh\" (UniqueName: \"kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh\") pod \"maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.235233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.235170 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:45.546791 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:08:45.546761 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bd0cf17_4e8a_4895_8626_e72c41d74500.slice/crio-dec64955f310b6d160e10a84f593ab642f2203a80c6da2d6717a0c54209daf4d WatchSource:0}: Error finding container dec64955f310b6d160e10a84f593ab642f2203a80c6da2d6717a0c54209daf4d: Status 404 returned error can't find the container with id dec64955f310b6d160e10a84f593ab642f2203a80c6da2d6717a0c54209daf4d Oct 30 06:08:45.546958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.546869 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57"] Oct 30 06:08:45.714572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:45.714508 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" event={"ID":"3bd0cf17-4e8a-4895-8626-e72c41d74500","Type":"ContainerStarted","Data":"dec64955f310b6d160e10a84f593ab642f2203a80c6da2d6717a0c54209daf4d"} Oct 30 06:08:46.145054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:46.145017 5565 patch_prober.go:29] interesting pod/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:08:46.145447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:46.145072 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:08:46.724269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:46.724230 5565 generic.go:356] "Generic (PLEG): container finished" podID="3bd0cf17-4e8a-4895-8626-e72c41d74500" containerID="f2583927bfa98bd4785c9027551ba2ad7917513d9c2f4b2d3fa346897590db7b" exitCode=0 Oct 30 06:08:46.724431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:46.724313 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" event={"ID":"3bd0cf17-4e8a-4895-8626-e72c41d74500","Type":"ContainerDied","Data":"f2583927bfa98bd4785c9027551ba2ad7917513d9c2f4b2d3fa346897590db7b"} Oct 30 06:08:47.505065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.505036 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:08:47.534982 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:08:47.534946 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:08:47.654065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654022 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654083 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654159 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654193 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654276 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654324 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654350 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.654460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654367 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.654460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654381 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654559 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:47.654758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654721 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.654931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654879 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket\") pod \"e13de906-e6b8-4b42-a065-d6b6540d0e24\" (UID: \"e13de906-e6b8-4b42-a065-d6b6540d0e24\") " Oct 30 06:08:47.655037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.654930 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.655385 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.655170 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.656741 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.656851 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.656876 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.656891 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e13de906-e6b8-4b42-a065-d6b6540d0e24-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.656905 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.657011 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:08:47.657128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.657069 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:47.657955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.657934 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "kube-api-access-kpsdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:47.658591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.658477 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo" pod "e13de906-e6b8-4b42-a065-d6b6540d0e24" (UID: "e13de906-e6b8-4b42-a065-d6b6540d0e24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:08:47.735001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.734917 5565 generic.go:356] "Generic (PLEG): container finished" podID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerID="e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d" exitCode=0 Oct 30 06:08:47.735001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.734982 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" Oct 30 06:08:47.735245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.735007 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" event={"ID":"e13de906-e6b8-4b42-a065-d6b6540d0e24","Type":"ContainerDied","Data":"e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d"} Oct 30 06:08:47.735245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.735073 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv" event={"ID":"e13de906-e6b8-4b42-a065-d6b6540d0e24","Type":"ContainerDied","Data":"f0fc8ecbd94b848debc16ec13dd28a28473b8212f4725f64cba84e7e7c2dce34"} Oct 30 06:08:47.735245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.735105 5565 scope.go:122] "RemoveContainer" containerID="e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d" Oct 30 06:08:47.750309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.750288 5565 scope.go:122] "RemoveContainer" containerID="e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d" Oct 30 06:08:47.751327 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:08:47.751298 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d\": container with ID starting with e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d not found: ID does not exist" containerID="e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d" Oct 30 06:08:47.751447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.751334 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d"} err="failed to get container status \"e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d\": rpc error: code = NotFound desc = could not find container \"e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d\": container with ID starting with e3d0dd673e33acf4a27c72469b08d05beb7be36d6ad5070ce105f21a3d69540d not found: ID does not exist" Oct 30 06:08:47.759180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.759154 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.759319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.759183 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpsdc\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-kube-api-access-kpsdc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.759319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.759197 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.759319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.759207 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e13de906-e6b8-4b42-a065-d6b6540d0e24-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:47.813031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.813000 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv"] Oct 30 06:08:47.817978 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.817950 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1dbaf6-openshift-default-845544477f-drvhv"] Oct 30 06:08:47.909041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:47.909012 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:48.063082 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.063046 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca\") pod \"3bd0cf17-4e8a-4895-8626-e72c41d74500\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " Oct 30 06:08:48.063253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.063085 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh\" (UniqueName: \"kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh\") pod \"3bd0cf17-4e8a-4895-8626-e72c41d74500\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " Oct 30 06:08:48.063309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.063297 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca\") pod \"3bd0cf17-4e8a-4895-8626-e72c41d74500\" (UID: \"3bd0cf17-4e8a-4895-8626-e72c41d74500\") " Oct 30 06:08:48.063419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.063395 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca" pod "3bd0cf17-4e8a-4895-8626-e72c41d74500" (UID: "3bd0cf17-4e8a-4895-8626-e72c41d74500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:48.063887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.063865 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:48.065255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.065234 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca" pod "3bd0cf17-4e8a-4895-8626-e72c41d74500" (UID: "3bd0cf17-4e8a-4895-8626-e72c41d74500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:48.065346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.065299 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh" pod "3bd0cf17-4e8a-4895-8626-e72c41d74500" (UID: "3bd0cf17-4e8a-4895-8626-e72c41d74500"). InnerVolumeSpecName "kube-api-access-m72wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:48.165169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.165133 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bd0cf17-4e8a-4895-8626-e72c41d74500-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:48.165169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.165169 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m72wh\" (UniqueName: \"kubernetes.io/projected/3bd0cf17-4e8a-4895-8626-e72c41d74500-kube-api-access-m72wh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:48.747097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.747065 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" Oct 30 06:08:48.747619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.747103 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57" event={"ID":"3bd0cf17-4e8a-4895-8626-e72c41d74500","Type":"ContainerDied","Data":"dec64955f310b6d160e10a84f593ab642f2203a80c6da2d6717a0c54209daf4d"} Oct 30 06:08:48.747619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:48.747157 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dec64955f310b6d160e10a84f593ab642f2203a80c6da2d6717a0c54209daf4d" Oct 30 06:08:49.568990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:49.568960 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" path="/var/lib/kubelet/pods/e13de906-e6b8-4b42-a065-d6b6540d0e24/volumes" Oct 30 06:08:50.400517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.400485 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j"] Oct 30 06:08:50.401048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.400787 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" podUID="cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" containerName="maas-api" containerID="cri-o://bdfdd98d246134d6d151dbd13191bdb96708a29561cfcdf92957cf25ac29d130" gracePeriod=30 Oct 30 06:08:50.879179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.879142 5565 generic.go:356] "Generic (PLEG): container finished" podID="cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" containerID="bdfdd98d246134d6d151dbd13191bdb96708a29561cfcdf92957cf25ac29d130" exitCode=0 Oct 30 06:08:50.879456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.879433 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" event={"ID":"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c","Type":"ContainerDied","Data":"bdfdd98d246134d6d151dbd13191bdb96708a29561cfcdf92957cf25ac29d130"} Oct 30 06:08:50.879603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.879490 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" event={"ID":"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c","Type":"ContainerDied","Data":"fed6c5d513a640bb8acfea8f189d92f7e515a2c45028143cfd2499ca4a59df3e"} Oct 30 06:08:50.879603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.879514 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fed6c5d513a640bb8acfea8f189d92f7e515a2c45028143cfd2499ca4a59df3e" Oct 30 06:08:50.909950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:50.903003 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:08:51.017266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.017237 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls\") pod \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " Oct 30 06:08:51.017459 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.017393 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs\") pod \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " Oct 30 06:08:51.017522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.017469 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca\") pod \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " Oct 30 06:08:51.017639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.017557 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg\" (UniqueName: \"kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg\") pod \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\" (UID: \"cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c\") " Oct 30 06:08:51.018308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.018267 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca" pod "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" (UID: "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:08:51.021058 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.021029 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls" pod "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" (UID: "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:51.021195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.021078 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs" pod "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" (UID: "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:08:51.021195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.021126 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg" pod "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" (UID: "cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c"). InnerVolumeSpecName "kube-api-access-9sxgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:08:51.119769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.119726 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.119769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.119770 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.119992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.119785 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sxgg\" (UniqueName: \"kubernetes.io/projected/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-kube-api-access-9sxgg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.119992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.119797 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:08:51.886869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.886782 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j" Oct 30 06:08:51.946483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.946452 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j"] Oct 30 06:08:51.960344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:51.960306 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-8de592-78fbdbbb59-gbc2j"] Oct 30 06:08:53.570074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:53.570033 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" path="/var/lib/kubelet/pods/cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c/volumes" Oct 30 06:08:59.392747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:59.392699 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 06:08:59.944084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:59.944029 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr"] Oct 30 06:08:59.944753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:08:59.944712 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" containerID="cri-o://5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4" gracePeriod=30 Oct 30 06:09:01.733252 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:01.733206 5565 patch_prober.go:29] interesting pod/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:01.733759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:01.733280 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:05.228634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.228569 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:09:05.256871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.256835 5565 generic.go:356] "Generic (PLEG): container finished" podID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerID="5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4" exitCode=0 Oct 30 06:09:05.257057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.256890 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" Oct 30 06:09:05.257057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.257006 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" event={"ID":"bc821543-7000-4f3a-bf81-e8d0e938a8cf","Type":"ContainerDied","Data":"5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4"} Oct 30 06:09:05.257177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.257057 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr" event={"ID":"bc821543-7000-4f3a-bf81-e8d0e938a8cf","Type":"ContainerDied","Data":"6588411dbf39a8feea6a5960c9fcbb6b9fb2303eb647cfd0732b3d844b35d1c5"} Oct 30 06:09:05.257177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.257080 5565 scope.go:122] "RemoveContainer" containerID="5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4" Oct 30 06:09:05.268605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.268585 5565 scope.go:122] "RemoveContainer" containerID="5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4" Oct 30 06:09:05.268904 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:09:05.268882 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4\": container with ID starting with 5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4 not found: ID does not exist" containerID="5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4" Oct 30 06:09:05.269043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.268910 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4"} err="failed to get container status \"5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4\": rpc error: code = NotFound desc = could not find container \"5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4\": container with ID starting with 5a545275e53bf9331fca495416f3d82051edf9a4b097ae404d166e862022c1c4 not found: ID does not exist" Oct 30 06:09:05.336052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336007 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336117 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336188 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336262 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336343 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.336587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336415 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336453 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336585 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336634 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336703 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:05.336823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336740 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy\") pod \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\" (UID: \"bc821543-7000-4f3a-bf81-e8d0e938a8cf\") " Oct 30 06:09:05.336906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336833 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.337006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.336980 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.337495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.337474 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.337679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.337659 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.337770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.337758 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.337850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.337840 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.337928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.337702 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.340392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.340368 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:05.341340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.341307 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:05.341450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.341433 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:05.341519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.341491 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75" pod "bc821543-7000-4f3a-bf81-e8d0e938a8cf" (UID: "bc821543-7000-4f3a-bf81-e8d0e938a8cf"). InnerVolumeSpecName "kube-api-access-sfr75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:05.475467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.475426 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.475467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.475457 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.475467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.475471 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfr75\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-kube-api-access-sfr75\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.475842 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.475485 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bc821543-7000-4f3a-bf81-e8d0e938a8cf-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.475842 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.475495 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bc821543-7000-4f3a-bf81-e8d0e938a8cf-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:05.610044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.610010 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr"] Oct 30 06:09:05.623919 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:05.623890 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8de592-openshift-default-544cdd5587-jctzr"] Oct 30 06:09:06.269768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:06.269737 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn"] Oct 30 06:09:06.270203 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:06.269954 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" containerID="cri-o://95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad" gracePeriod=30 Oct 30 06:09:07.570337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:07.570307 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" path="/var/lib/kubelet/pods/bc821543-7000-4f3a-bf81-e8d0e938a8cf/volumes" Oct 30 06:09:07.902076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:07.901992 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6"] Oct 30 06:09:07.902241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:07.902188 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" containerID="cri-o://6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1" gracePeriod=30 Oct 30 06:09:09.985879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:09.985844 5565 patch_prober.go:29] interesting pod/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:09:09.986358 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:09.985907 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:09:11.575925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.575897 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 06:09:11.597733 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.597707 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns"] Oct 30 06:09:11.611949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.611919 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-482468-84dns"] Oct 30 06:09:11.695672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.695642 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.695834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.695693 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.695834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.695758 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.695909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.695835 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.695959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.695909 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.696053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696035 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.696155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696073 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.696155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696086 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.696155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696083 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.696344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696164 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.696344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696235 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.696450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696331 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket\") pod \"946e069a-987f-4ede-a2d4-cd376c83d4e7\" (UID: \"946e069a-987f-4ede-a2d4-cd376c83d4e7\") " Oct 30 06:09:11.696500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696456 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:11.696603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.696581 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.697255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.697227 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.697379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.697256 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.697379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.697274 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.697379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.697288 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.697379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.697303 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/946e069a-987f-4ede-a2d4-cd376c83d4e7-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.698007 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.697980 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "kube-api-access-r6jkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:11.698290 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.698260 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:11.698290 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.698278 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:11.699238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.699222 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo" pod "946e069a-987f-4ede-a2d4-cd376c83d4e7" (UID: "946e069a-987f-4ede-a2d4-cd376c83d4e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:11.798694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.798603 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.798694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.798635 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.798694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.798649 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6jkd\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-kube-api-access-r6jkd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:11.798694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:11.798661 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/946e069a-987f-4ede-a2d4-cd376c83d4e7-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:12.315844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.315809 5565 generic.go:356] "Generic (PLEG): container finished" podID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerID="95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad" exitCode=0 Oct 30 06:09:12.316019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.315855 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" Oct 30 06:09:12.316019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.315888 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" event={"ID":"946e069a-987f-4ede-a2d4-cd376c83d4e7","Type":"ContainerDied","Data":"95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad"} Oct 30 06:09:12.316019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.315938 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn" event={"ID":"946e069a-987f-4ede-a2d4-cd376c83d4e7","Type":"ContainerDied","Data":"e19173b0e7dd07041c17a886d35e79098ad76957c8c1ae53500843a793ace729"} Oct 30 06:09:12.316019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.315958 5565 scope.go:122] "RemoveContainer" containerID="95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad" Oct 30 06:09:12.358114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.358086 5565 scope.go:122] "RemoveContainer" containerID="95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad" Oct 30 06:09:12.358445 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:09:12.358417 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad\": container with ID starting with 95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad not found: ID does not exist" containerID="95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad" Oct 30 06:09:12.358595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.358452 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad"} err="failed to get container status \"95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad\": rpc error: code = NotFound desc = could not find container \"95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad\": container with ID starting with 95d17aa8246f175fb41cd6a91bde6168100cbc641978f79008ce736a1118c5ad not found: ID does not exist" Oct 30 06:09:12.393184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.393159 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn"] Oct 30 06:09:12.407128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:12.407103 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-1d8c2f-openshift-default-5bc9c48c9-wmrsn"] Oct 30 06:09:13.222500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.222472 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 06:09:13.320363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320323 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.320568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320378 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.320660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320605 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.320660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320647 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.320783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320760 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.320847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320830 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.320954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.320934 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.321069 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.321111 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.321131 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.321214 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket\") pod \"e12372fc-9176-4c18-885f-d77075d6f514\" (UID: \"e12372fc-9176-4c18-885f-d77075d6f514\") " Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.321659 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.321867 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.322068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.322027 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:13.322563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.322273 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.322563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.322294 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.322563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.322307 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.322563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.322342 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.322563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.322355 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e12372fc-9176-4c18-885f-d77075d6f514-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.323727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.323692 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "kube-api-access-dnzgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:13.323976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.323950 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:13.325267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.325243 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:13.325477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.325460 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy" pod "e12372fc-9176-4c18-885f-d77075d6f514" (UID: "e12372fc-9176-4c18-885f-d77075d6f514"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:13.330759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.330729 5565 generic.go:356] "Generic (PLEG): container finished" podID="e12372fc-9176-4c18-885f-d77075d6f514" containerID="6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1" exitCode=0 Oct 30 06:09:13.330924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.330772 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" Oct 30 06:09:13.330924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.330803 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" event={"ID":"e12372fc-9176-4c18-885f-d77075d6f514","Type":"ContainerDied","Data":"6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1"} Oct 30 06:09:13.330924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.330852 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6" event={"ID":"e12372fc-9176-4c18-885f-d77075d6f514","Type":"ContainerDied","Data":"11c37270dff4a1bdbe976e57d71d05ed3cd7c031ee8e6c66a956080fad115eda"} Oct 30 06:09:13.330924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.330883 5565 scope.go:122] "RemoveContainer" containerID="6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1" Oct 30 06:09:13.362066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.362050 5565 scope.go:122] "RemoveContainer" containerID="6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1" Oct 30 06:09:13.362339 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:09:13.362317 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1\": container with ID starting with 6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1 not found: ID does not exist" containerID="6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1" Oct 30 06:09:13.362387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.362346 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1"} err="failed to get container status \"6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1\": rpc error: code = NotFound desc = could not find container \"6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1\": container with ID starting with 6317208a7f698b2a238c1d9bf5e696e12058bbecddf95db2eeee57c7e7999af1 not found: ID does not exist" Oct 30 06:09:13.399350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.399307 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6"] Oct 30 06:09:13.404268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.404247 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-rlkc6"] Oct 30 06:09:13.424364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.424316 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e12372fc-9176-4c18-885f-d77075d6f514-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.424364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.424362 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.424623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.424375 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e12372fc-9176-4c18-885f-d77075d6f514-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.424623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.424389 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnzgr\" (UniqueName: \"kubernetes.io/projected/e12372fc-9176-4c18-885f-d77075d6f514-kube-api-access-dnzgr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:13.570454 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.570422 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef" path="/var/lib/kubelet/pods/8dc5d1f2-b788-4f6d-aca6-0f47da99b5ef/volumes" Oct 30 06:09:13.570962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.570947 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" path="/var/lib/kubelet/pods/946e069a-987f-4ede-a2d4-cd376c83d4e7/volumes" Oct 30 06:09:13.571430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:13.571416 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e12372fc-9176-4c18-885f-d77075d6f514" path="/var/lib/kubelet/pods/e12372fc-9176-4c18-885f-d77075d6f514/volumes" Oct 30 06:09:17.534182 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:09:17.534139 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:09:19.128928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.128894 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq"] Oct 30 06:09:19.129620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129605 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" Oct 30 06:09:19.129691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129621 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" Oct 30 06:09:19.129691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129639 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" containerName="maas-api" Oct 30 06:09:19.129691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129644 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" containerName="maas-api" Oct 30 06:09:19.129691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129654 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" Oct 30 06:09:19.129691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129685 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" Oct 30 06:09:19.129862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129705 5565 cpu_manager.go:414] "containerMap: removing container" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" Oct 30 06:09:19.129862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129710 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" Oct 30 06:09:19.129862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129719 5565 cpu_manager.go:414] "containerMap: removing container" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" Oct 30 06:09:19.129862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129724 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" Oct 30 06:09:19.129862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129765 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3bd0cf17-4e8a-4895-8626-e72c41d74500" containerName="revoke-keys" Oct 30 06:09:19.129862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129769 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bd0cf17-4e8a-4895-8626-e72c41d74500" containerName="revoke-keys" Oct 30 06:09:19.130048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129877 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bd0cf17-4e8a-4895-8626-e72c41d74500" containerName="revoke-keys" Oct 30 06:09:19.130048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129903 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc821543-7000-4f3a-bf81-e8d0e938a8cf" containerName="istio-proxy" Oct 30 06:09:19.130048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129912 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc7de6c1-a8ab-4b17-ab4f-e03638bb7f9c" containerName="maas-api" Oct 30 06:09:19.130048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129917 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e12372fc-9176-4c18-885f-d77075d6f514" containerName="istio-proxy" Oct 30 06:09:19.130048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129925 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="946e069a-987f-4ede-a2d4-cd376c83d4e7" containerName="istio-proxy" Oct 30 06:09:19.130048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.129931 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13de906-e6b8-4b42-a065-d6b6540d0e24" containerName="istio-proxy" Oct 30 06:09:19.134873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.134854 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.154570 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.154543 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq"] Oct 30 06:09:19.240421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.233701 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.247239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.247186 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.247507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.247483 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dpw\" (UniqueName: \"kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.247635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.240130 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:09:19.247635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.240075 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:09:19.349350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.349311 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.349563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.349417 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.349697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.349679 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dpw\" (UniqueName: \"kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.350122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.350094 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.351907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.351881 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.375137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.375108 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dpw\" (UniqueName: \"kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw\") pod \"maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.449949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.449875 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:19.759840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:19.759816 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq"] Oct 30 06:09:19.760407 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:09:19.760377 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1b7b03d_2198_4a5f_964c_2a5f55d12261.slice/crio-dece223afdc2e16ee4377d396cd472e840a2df416477f1aaab6fd5b117919412 WatchSource:0}: Error finding container dece223afdc2e16ee4377d396cd472e840a2df416477f1aaab6fd5b117919412: Status 404 returned error can't find the container with id dece223afdc2e16ee4377d396cd472e840a2df416477f1aaab6fd5b117919412 Oct 30 06:09:20.393624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:20.393591 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" event={"ID":"d1b7b03d-2198-4a5f-964c-2a5f55d12261","Type":"ContainerStarted","Data":"88477d2c46eb2454b0c1296b836cd0022294ab0d783c5b6da7f8702ae3b847c7"} Oct 30 06:09:20.394000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:20.393644 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" event={"ID":"d1b7b03d-2198-4a5f-964c-2a5f55d12261","Type":"ContainerStarted","Data":"dece223afdc2e16ee4377d396cd472e840a2df416477f1aaab6fd5b117919412"} Oct 30 06:09:20.440138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:20.440038 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" podStartSLOduration=1.440021126 podStartE2EDuration="1.440021126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:09:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:09:20.437106008 +0000 UTC m=+2267.012793086" watchObservedRunningTime="2026-10-30 06:09:20.440021126 +0000 UTC m=+2267.015708180" Oct 30 06:09:21.402361 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:21.402331 5565 generic.go:356] "Generic (PLEG): container finished" podID="d1b7b03d-2198-4a5f-964c-2a5f55d12261" containerID="88477d2c46eb2454b0c1296b836cd0022294ab0d783c5b6da7f8702ae3b847c7" exitCode=0 Oct 30 06:09:21.402788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:21.402410 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" event={"ID":"d1b7b03d-2198-4a5f-964c-2a5f55d12261","Type":"ContainerDied","Data":"88477d2c46eb2454b0c1296b836cd0022294ab0d783c5b6da7f8702ae3b847c7"} Oct 30 06:09:21.732096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:21.731985 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57"] Oct 30 06:09:21.738386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:21.738356 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-f28e2d-8cs57"] Oct 30 06:09:22.657836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.657799 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:22.754172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.754143 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca\") pod \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " Oct 30 06:09:22.754441 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.754423 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw\" (UniqueName: \"kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw\") pod \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " Oct 30 06:09:22.754514 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.754445 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca" pod "d1b7b03d-2198-4a5f-964c-2a5f55d12261" (UID: "d1b7b03d-2198-4a5f-964c-2a5f55d12261"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:22.754607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.754588 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca\") pod \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\" (UID: \"d1b7b03d-2198-4a5f-964c-2a5f55d12261\") " Oct 30 06:09:22.755112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.755093 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:22.756504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.756484 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca" pod "d1b7b03d-2198-4a5f-964c-2a5f55d12261" (UID: "d1b7b03d-2198-4a5f-964c-2a5f55d12261"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:22.756639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.756501 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw" pod "d1b7b03d-2198-4a5f-964c-2a5f55d12261" (UID: "d1b7b03d-2198-4a5f-964c-2a5f55d12261"). InnerVolumeSpecName "kube-api-access-65dpw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:22.857069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.857042 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65dpw\" (UniqueName: \"kubernetes.io/projected/d1b7b03d-2198-4a5f-964c-2a5f55d12261-kube-api-access-65dpw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:22.857069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:22.857071 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d1b7b03d-2198-4a5f-964c-2a5f55d12261-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:23.419556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:23.419496 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" Oct 30 06:09:23.419765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:23.419559 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq" event={"ID":"d1b7b03d-2198-4a5f-964c-2a5f55d12261","Type":"ContainerDied","Data":"dece223afdc2e16ee4377d396cd472e840a2df416477f1aaab6fd5b117919412"} Oct 30 06:09:23.419765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:23.419638 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dece223afdc2e16ee4377d396cd472e840a2df416477f1aaab6fd5b117919412" Oct 30 06:09:23.575751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:23.575717 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bd0cf17-4e8a-4895-8626-e72c41d74500" path="/var/lib/kubelet/pods/3bd0cf17-4e8a-4895-8626-e72c41d74500/volumes" Oct 30 06:09:30.646521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:30.646483 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz"] Oct 30 06:09:30.646997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:30.646770 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" containerID="cri-o://dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8" gracePeriod=30 Oct 30 06:09:35.975490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:35.975462 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:09:36.014054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014021 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014103 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014192 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014212 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014348 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014247 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014407 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014359 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014459 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014421 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014484 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014513 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:36.014640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014588 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:36.014640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014630 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy\") pod \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\" (UID: \"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891\") " Oct 30 06:09:36.014949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.014924 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:36.015463 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.015441 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.015605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.015471 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.015605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.015486 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.015605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.015591 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:36.015771 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.015658 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:36.017004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.016966 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:09:36.017004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.016995 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "kube-api-access-7bj95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:36.017162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.017024 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:09:36.017162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.017065 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy" pod "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" (UID: "2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:09:36.117214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.117178 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.117214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.117208 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.117214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.117220 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.117451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.117230 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bj95\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-kube-api-access-7bj95\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.117451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.117239 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.117451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.117247 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:09:36.538590 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.538523 5565 generic.go:356] "Generic (PLEG): container finished" podID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerID="dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8" exitCode=0 Oct 30 06:09:36.538755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.538622 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" Oct 30 06:09:36.538755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.538565 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" event={"ID":"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891","Type":"ContainerDied","Data":"dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8"} Oct 30 06:09:36.538755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.538722 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz" event={"ID":"2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891","Type":"ContainerDied","Data":"b36bbc80327191f9016872b3d62278e55f9dda1d6e24a516b370d7b06c268b7a"} Oct 30 06:09:36.538755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.538746 5565 scope.go:122] "RemoveContainer" containerID="dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8" Oct 30 06:09:36.551507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.551488 5565 scope.go:122] "RemoveContainer" containerID="dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8" Oct 30 06:09:36.551834 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:09:36.551815 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8\": container with ID starting with dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8 not found: ID does not exist" containerID="dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8" Oct 30 06:09:36.551891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.551841 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8"} err="failed to get container status \"dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8\": rpc error: code = NotFound desc = could not find container \"dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8\": container with ID starting with dba797a97a68697527d7d2e0f0242e16ee946d551fbdbedb6a812e643c7cdad8 not found: ID does not exist" Oct 30 06:09:36.599275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.599250 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz"] Oct 30 06:09:36.612502 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:36.612476 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-38cfad-openshift-default-76bc4b755-c7ffz"] Oct 30 06:09:37.585935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:37.585902 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" path="/var/lib/kubelet/pods/2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891/volumes" Oct 30 06:09:38.184072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:38.184039 5565 scope.go:122] "RemoveContainer" containerID="bdfdd98d246134d6d151dbd13191bdb96708a29561cfcdf92957cf25ac29d130" Oct 30 06:09:47.573625 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:09:47.573588 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:09:49.726873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:49.726830 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:09:56.411110 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:56.411079 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq"] Oct 30 06:09:56.418649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:56.418626 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-d917a1-j78tq"] Oct 30 06:09:57.569292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:57.569256 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1b7b03d-2198-4a5f-964c-2a5f55d12261" path="/var/lib/kubelet/pods/d1b7b03d-2198-4a5f-964c-2a5f55d12261/volumes" Oct 30 06:09:59.434609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.434576 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f"] Oct 30 06:09:59.435054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.434841 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" podUID="5cd96d8e-a1fc-4b3d-9192-0994693701c7" containerName="maas-api" containerID="cri-o://0a7bd610e6059a1f92e6ae81aec7dae15290997fd00143df0d6c36656bdc8028" gracePeriod=30 Oct 30 06:09:59.785171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.785136 5565 generic.go:356] "Generic (PLEG): container finished" podID="5cd96d8e-a1fc-4b3d-9192-0994693701c7" containerID="0a7bd610e6059a1f92e6ae81aec7dae15290997fd00143df0d6c36656bdc8028" exitCode=0 Oct 30 06:09:59.785364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.785225 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" event={"ID":"5cd96d8e-a1fc-4b3d-9192-0994693701c7","Type":"ContainerDied","Data":"0a7bd610e6059a1f92e6ae81aec7dae15290997fd00143df0d6c36656bdc8028"} Oct 30 06:09:59.785364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.785276 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" event={"ID":"5cd96d8e-a1fc-4b3d-9192-0994693701c7","Type":"ContainerDied","Data":"e69a12cc040c0dbb4154cbe2e4b7dbb4ba8490668afe603ddfe5672bd8fbeac6"} Oct 30 06:09:59.785364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.785301 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e69a12cc040c0dbb4154cbe2e4b7dbb4ba8490668afe603ddfe5672bd8fbeac6" Oct 30 06:09:59.794740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.794715 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:09:59.930955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.929236 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs\") pod \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " Oct 30 06:09:59.930955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.929307 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca\") pod \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " Oct 30 06:09:59.930955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.929333 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx\" (UniqueName: \"kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx\") pod \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " Oct 30 06:09:59.930955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.929557 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls\") pod \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\" (UID: \"5cd96d8e-a1fc-4b3d-9192-0994693701c7\") " Oct 30 06:09:59.947562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.943039 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca" pod "5cd96d8e-a1fc-4b3d-9192-0994693701c7" (UID: "5cd96d8e-a1fc-4b3d-9192-0994693701c7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:09:59.959550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.955961 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls" pod "5cd96d8e-a1fc-4b3d-9192-0994693701c7" (UID: "5cd96d8e-a1fc-4b3d-9192-0994693701c7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:59.959550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.957964 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs" pod "5cd96d8e-a1fc-4b3d-9192-0994693701c7" (UID: "5cd96d8e-a1fc-4b3d-9192-0994693701c7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:09:59.959550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:09:59.958685 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx" pod "5cd96d8e-a1fc-4b3d-9192-0994693701c7" (UID: "5cd96d8e-a1fc-4b3d-9192-0994693701c7"). InnerVolumeSpecName "kube-api-access-xfpsx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:00.031576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.031540 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:00.031576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.031570 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd96d8e-a1fc-4b3d-9192-0994693701c7-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:00.031576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.031580 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd96d8e-a1fc-4b3d-9192-0994693701c7-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:00.031873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.031626 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfpsx\" (UniqueName: \"kubernetes.io/projected/5cd96d8e-a1fc-4b3d-9192-0994693701c7-kube-api-access-xfpsx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:00.791915 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.791885 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f" Oct 30 06:10:00.851685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.851650 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f"] Oct 30 06:10:00.865477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:00.865449 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-7dfcb7-5d888d546f-5255f"] Oct 30 06:10:01.570402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:01.570365 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd96d8e-a1fc-4b3d-9192-0994693701c7" path="/var/lib/kubelet/pods/5cd96d8e-a1fc-4b3d-9192-0994693701c7/volumes" Oct 30 06:10:03.596057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:03.596021 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5"] Oct 30 06:10:03.596557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:03.596285 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" containerID="cri-o://7f6261a4f063267eb28d746b394a3b2ef3a91d3ef4dd834378d9089d5fcb2761" gracePeriod=30 Oct 30 06:10:08.005076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.005038 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl"] Oct 30 06:10:08.005604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.005369 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" containerID="cri-o://202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d" gracePeriod=30 Oct 30 06:10:08.313807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.313729 5565 patch_prober.go:29] interesting pod/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:10:08.313807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.313797 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:10:08.867481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.867406 5565 generic.go:356] "Generic (PLEG): container finished" podID="60308810-612a-4400-b5d3-eeea99079add" containerID="7f6261a4f063267eb28d746b394a3b2ef3a91d3ef4dd834378d9089d5fcb2761" exitCode=0 Oct 30 06:10:08.867649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.867478 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" event={"ID":"60308810-612a-4400-b5d3-eeea99079add","Type":"ContainerDied","Data":"7f6261a4f063267eb28d746b394a3b2ef3a91d3ef4dd834378d9089d5fcb2761"} Oct 30 06:10:08.867649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.867550 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" event={"ID":"60308810-612a-4400-b5d3-eeea99079add","Type":"ContainerDied","Data":"8c43b82220599eda78d4327dfbcb21ef5c326dfe4318e07a11ea3b225c316f10"} Oct 30 06:10:08.867649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.867567 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c43b82220599eda78d4327dfbcb21ef5c326dfe4318e07a11ea3b225c316f10" Oct 30 06:10:08.867649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.867581 5565 scope.go:122] "RemoveContainer" containerID="6265d208d55a75f7ccb9bd4abd198bca005775b47d4581696816740c488960a6" Oct 30 06:10:08.882595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.882571 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:10:08.936447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936417 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936452 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936573 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936600 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936679 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936733 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936768 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.936976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936781 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936819 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936884 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.936976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936905 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert\") pod \"60308810-612a-4400-b5d3-eeea99079add\" (UID: \"60308810-612a-4400-b5d3-eeea99079add\") " Oct 30 06:10:08.936976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936965 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.937299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.936997 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.937387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.937357 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:08.938171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.938146 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:08.939845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.939584 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:08.939845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.939268 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "kube-api-access-5s269". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:08.939845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.939605 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:08.939845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.939650 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:08.939845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.939662 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/60308810-612a-4400-b5d3-eeea99079add-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:08.944678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.944651 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:08.944817 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.944688 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:08.945434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:08.945409 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo" pod "60308810-612a-4400-b5d3-eeea99079add" (UID: "60308810-612a-4400-b5d3-eeea99079add"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:09.041719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.041678 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5s269\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-kube-api-access-5s269\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.041719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.041718 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/60308810-612a-4400-b5d3-eeea99079add-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.042141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.041733 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/60308810-612a-4400-b5d3-eeea99079add-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.042141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.041745 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/60308810-612a-4400-b5d3-eeea99079add-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:09.882495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.882400 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5" Oct 30 06:10:09.993909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.993867 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5"] Oct 30 06:10:09.998283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:09.998253 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d917a1-openshift-default-f55bbcf9f-zpnw5"] Oct 30 06:10:11.568914 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:11.568879 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60308810-612a-4400-b5d3-eeea99079add" path="/var/lib/kubelet/pods/60308810-612a-4400-b5d3-eeea99079add/volumes" Oct 30 06:10:13.287323 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.287259 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:10:13.420280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420243 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420285 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420311 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420371 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420467 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420503 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420664 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.420804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420663 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.420804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420685 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.420804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420727 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.421010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420813 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.421010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420834 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:13.421010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.420860 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29\") pod \"2a58a055-136e-4160-b118-b26a3674b80f\" (UID: \"2a58a055-136e-4160-b118-b26a3674b80f\") " Oct 30 06:10:13.421156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.421048 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.421603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.421575 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.421603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.421603 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.421802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.421618 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2a58a055-136e-4160-b118-b26a3674b80f-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.421802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.421632 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.421802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.421646 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.422761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.422733 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:10:13.422863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.422827 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:10:13.422906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.422882 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:13.423026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.423004 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29" pod "2a58a055-136e-4160-b118-b26a3674b80f" (UID: "2a58a055-136e-4160-b118-b26a3674b80f"). InnerVolumeSpecName "kube-api-access-lxl29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:13.523415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.523379 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2a58a055-136e-4160-b118-b26a3674b80f-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.523415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.523409 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.523415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.523418 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2a58a055-136e-4160-b118-b26a3674b80f-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.523724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.523429 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxl29\" (UniqueName: \"kubernetes.io/projected/2a58a055-136e-4160-b118-b26a3674b80f-kube-api-access-lxl29\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:13.932104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.932013 5565 generic.go:356] "Generic (PLEG): container finished" podID="2a58a055-136e-4160-b118-b26a3674b80f" containerID="202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d" exitCode=0 Oct 30 06:10:13.932104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.932070 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" Oct 30 06:10:13.932369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.932098 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" event={"ID":"2a58a055-136e-4160-b118-b26a3674b80f","Type":"ContainerDied","Data":"202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d"} Oct 30 06:10:13.932369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.932186 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl" event={"ID":"2a58a055-136e-4160-b118-b26a3674b80f","Type":"ContainerDied","Data":"4d8405edee82b974a2c6b5470638f3762ce79bcbb908f2d0b2752daf4f2986b5"} Oct 30 06:10:13.932369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.932215 5565 scope.go:122] "RemoveContainer" containerID="202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d" Oct 30 06:10:13.943139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.943117 5565 scope.go:122] "RemoveContainer" containerID="202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d" Oct 30 06:10:13.943485 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:10:13.943457 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d\": container with ID starting with 202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d not found: ID does not exist" containerID="202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d" Oct 30 06:10:13.943591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:13.943486 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d"} err="failed to get container status \"202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d\": rpc error: code = NotFound desc = could not find container \"202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d\": container with ID starting with 202c1f1ac0af6da48a8c908bd1c8e4364b0797f6fcfa02efacdfc0cbccc0688d not found: ID does not exist" Oct 30 06:10:14.006350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:14.006316 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl"] Oct 30 06:10:14.020151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:14.020121 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-7dfcb7-openshift-default-679f85b559-c2nbl"] Oct 30 06:10:15.569228 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:15.569194 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a58a055-136e-4160-b118-b26a3674b80f" path="/var/lib/kubelet/pods/2a58a055-136e-4160-b118-b26a3674b80f/volumes" Oct 30 06:10:17.534403 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:10:17.534368 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:10:37.415610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.415570 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5"] Oct 30 06:10:37.416251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416235 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5cd96d8e-a1fc-4b3d-9192-0994693701c7" containerName="maas-api" Oct 30 06:10:37.416302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416253 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd96d8e-a1fc-4b3d-9192-0994693701c7" containerName="maas-api" Oct 30 06:10:37.416302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416293 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" Oct 30 06:10:37.416302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416298 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416314 5565 cpu_manager.go:414] "containerMap: removing container" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416319 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416332 5565 cpu_manager.go:414] "containerMap: removing container" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416346 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416362 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d1b7b03d-2198-4a5f-964c-2a5f55d12261" containerName="revoke-keys" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416366 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1b7b03d-2198-4a5f-964c-2a5f55d12261" containerName="revoke-keys" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416375 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" Oct 30 06:10:37.416414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416380 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" Oct 30 06:10:37.416695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416475 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" Oct 30 06:10:37.416695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416485 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1b7b03d-2198-4a5f-964c-2a5f55d12261" containerName="revoke-keys" Oct 30 06:10:37.416695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416494 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a96ca1a-bfd3-42dd-a2a2-718bfc1eb891" containerName="istio-proxy" Oct 30 06:10:37.416695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416501 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd96d8e-a1fc-4b3d-9192-0994693701c7" containerName="maas-api" Oct 30 06:10:37.416695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.416511 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a58a055-136e-4160-b118-b26a3674b80f" containerName="istio-proxy" Oct 30 06:10:37.419641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.419622 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.440326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.440298 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5"] Oct 30 06:10:37.449371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.449344 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:37.449521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.449412 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:10:37.459148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.459123 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.459334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.459311 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q59tc\" (UniqueName: \"kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.459475 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.459461 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.560596 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.560566 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.560740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.560679 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q59tc\" (UniqueName: \"kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.560785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.560744 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.561200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.561178 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.563062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.563040 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.585828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.585803 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q59tc\" (UniqueName: \"kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc\") pod \"maas-api-revoke-sub-87691408-4zhx5\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:37.734469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:37.734398 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:38.066852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:38.066815 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5"] Oct 30 06:10:38.067310 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:10:38.067281 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdf5604_7b91_4885_9224_722b777e9f1a.slice/crio-6a31578d4ed37b56cd90909bff2bda7a157328cc7466144bd9d6bc38b7f36485 WatchSource:0}: Error finding container 6a31578d4ed37b56cd90909bff2bda7a157328cc7466144bd9d6bc38b7f36485: Status 404 returned error can't find the container with id 6a31578d4ed37b56cd90909bff2bda7a157328cc7466144bd9d6bc38b7f36485 Oct 30 06:10:38.241697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:38.241658 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" event={"ID":"bfdf5604-7b91-4885-9224-722b777e9f1a","Type":"ContainerStarted","Data":"6a31578d4ed37b56cd90909bff2bda7a157328cc7466144bd9d6bc38b7f36485"} Oct 30 06:10:38.243712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:38.243687 5565 scope.go:122] "RemoveContainer" containerID="7f6261a4f063267eb28d746b394a3b2ef3a91d3ef4dd834378d9089d5fcb2761" Oct 30 06:10:38.254836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:38.254813 5565 scope.go:122] "RemoveContainer" containerID="0a7bd610e6059a1f92e6ae81aec7dae15290997fd00143df0d6c36656bdc8028" Oct 30 06:10:39.251618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:39.251506 5565 generic.go:356] "Generic (PLEG): container finished" podID="bfdf5604-7b91-4885-9224-722b777e9f1a" containerID="a3507f07071327f13f89cd7a839e79b9ead93a871b14112f700736f680d66553" exitCode=0 Oct 30 06:10:39.251618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:39.251558 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" event={"ID":"bfdf5604-7b91-4885-9224-722b777e9f1a","Type":"ContainerDied","Data":"a3507f07071327f13f89cd7a839e79b9ead93a871b14112f700736f680d66553"} Oct 30 06:10:40.427387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.427357 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:40.500886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.500853 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca\") pod \"bfdf5604-7b91-4885-9224-722b777e9f1a\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " Oct 30 06:10:40.501073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.500964 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca\") pod \"bfdf5604-7b91-4885-9224-722b777e9f1a\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " Oct 30 06:10:40.501142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.501072 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc\" (UniqueName: \"kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc\") pod \"bfdf5604-7b91-4885-9224-722b777e9f1a\" (UID: \"bfdf5604-7b91-4885-9224-722b777e9f1a\") " Oct 30 06:10:40.501741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.501710 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca" pod "bfdf5604-7b91-4885-9224-722b777e9f1a" (UID: "bfdf5604-7b91-4885-9224-722b777e9f1a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:40.503193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.503167 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca" pod "bfdf5604-7b91-4885-9224-722b777e9f1a" (UID: "bfdf5604-7b91-4885-9224-722b777e9f1a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:40.503314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.503218 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc" pod "bfdf5604-7b91-4885-9224-722b777e9f1a" (UID: "bfdf5604-7b91-4885-9224-722b777e9f1a"). InnerVolumeSpecName "kube-api-access-q59tc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:40.603642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.603606 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.603642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.603639 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bfdf5604-7b91-4885-9224-722b777e9f1a-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:40.603839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:40.603654 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q59tc\" (UniqueName: \"kubernetes.io/projected/bfdf5604-7b91-4885-9224-722b777e9f1a-kube-api-access-q59tc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:41.268965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:41.268925 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" event={"ID":"bfdf5604-7b91-4885-9224-722b777e9f1a","Type":"ContainerDied","Data":"6a31578d4ed37b56cd90909bff2bda7a157328cc7466144bd9d6bc38b7f36485"} Oct 30 06:10:41.269170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:41.268983 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a31578d4ed37b56cd90909bff2bda7a157328cc7466144bd9d6bc38b7f36485" Oct 30 06:10:41.269170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:41.268942 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5" Oct 30 06:10:47.535143 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:10:47.535060 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:10:47.851257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.851160 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7"] Oct 30 06:10:47.852276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.852252 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bfdf5604-7b91-4885-9224-722b777e9f1a" containerName="revoke-keys" Oct 30 06:10:47.852276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.852275 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bfdf5604-7b91-4885-9224-722b777e9f1a" containerName="revoke-keys" Oct 30 06:10:47.852468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.852450 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bfdf5604-7b91-4885-9224-722b777e9f1a" containerName="revoke-keys" Oct 30 06:10:47.852512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.852470 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="60308810-612a-4400-b5d3-eeea99079add" containerName="istio-proxy" Oct 30 06:10:47.856709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.856687 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.873732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.873703 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7"] Oct 30 06:10:47.882516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.882486 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.882731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.882557 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:47.882731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.882678 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.883044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.882744 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqk7\" (UniqueName: \"kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.883201 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.882763 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:10:47.983917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.983880 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.984119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.983969 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.984119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.984005 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqk7\" (UniqueName: \"kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.984629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.984604 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:47.986276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:47.986254 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:48.009556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:48.009485 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqk7\" (UniqueName: \"kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7\") pod \"maas-api-revoke-sub-663d6682-l5nn7\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:48.171664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:48.171571 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:48.484028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:48.483994 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7"] Oct 30 06:10:48.484866 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:10:48.484835 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc94cc0f9_ead4_4be9_9e19_9ed3eb53d6d4.slice/crio-5a99a0d395ef5172154d5372749c475a395c4bc491f3a2e1b358f01a2949e007 WatchSource:0}: Error finding container 5a99a0d395ef5172154d5372749c475a395c4bc491f3a2e1b358f01a2949e007: Status 404 returned error can't find the container with id 5a99a0d395ef5172154d5372749c475a395c4bc491f3a2e1b358f01a2949e007 Oct 30 06:10:49.334154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:49.334116 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" event={"ID":"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4","Type":"ContainerStarted","Data":"2982141c01f810595dae3ead09af58689de516e0f38de0e511ebbc06f68d7efe"} Oct 30 06:10:49.334673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:49.334180 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" event={"ID":"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4","Type":"ContainerStarted","Data":"5a99a0d395ef5172154d5372749c475a395c4bc491f3a2e1b358f01a2949e007"} Oct 30 06:10:49.381439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:49.381393 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" podStartSLOduration=2.381378606 podStartE2EDuration="2.381378606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:10:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:10:49.378747708 +0000 UTC m=+2355.954434762" watchObservedRunningTime="2026-10-30 06:10:49.381378606 +0000 UTC m=+2355.957065659" Oct 30 06:10:50.343676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:50.343639 5565 generic.go:356] "Generic (PLEG): container finished" podID="c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" containerID="2982141c01f810595dae3ead09af58689de516e0f38de0e511ebbc06f68d7efe" exitCode=0 Oct 30 06:10:50.344121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:50.343709 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" event={"ID":"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4","Type":"ContainerDied","Data":"2982141c01f810595dae3ead09af58689de516e0f38de0e511ebbc06f68d7efe"} Oct 30 06:10:51.518267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.518245 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:51.638042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.637965 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca\") pod \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " Oct 30 06:10:51.638281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.638253 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca" pod "c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" (UID: "c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:10:51.638352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.638291 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca\") pod \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " Oct 30 06:10:51.638410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.638352 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7\" (UniqueName: \"kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7\") pod \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\" (UID: \"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4\") " Oct 30 06:10:51.639018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.638991 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.640994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.640964 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca" pod "c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" (UID: "c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:10:51.661052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.661019 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7" pod "c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" (UID: "c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4"). InnerVolumeSpecName "kube-api-access-tsqk7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:10:51.741197 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.741168 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:51.741197 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:51.741196 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqk7\" (UniqueName: \"kubernetes.io/projected/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4-kube-api-access-tsqk7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:10:52.362013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:52.361980 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" Oct 30 06:10:52.362013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:52.362007 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7" event={"ID":"c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4","Type":"ContainerDied","Data":"5a99a0d395ef5172154d5372749c475a395c4bc491f3a2e1b358f01a2949e007"} Oct 30 06:10:52.371697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:52.362108 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a99a0d395ef5172154d5372749c475a395c4bc491f3a2e1b358f01a2949e007" Oct 30 06:10:58.842278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.842240 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl"] Oct 30 06:10:58.842985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.842969 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" containerName="revoke-keys" Oct 30 06:10:58.842985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.842986 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" containerName="revoke-keys" Oct 30 06:10:58.843147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.843131 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" containerName="revoke-keys" Oct 30 06:10:58.846556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.846508 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:58.873340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.873309 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl"] Oct 30 06:10:58.876030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.876002 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:10:58.876181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.876114 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:10:58.943295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.943260 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx5v8\" (UniqueName: \"kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:58.943469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.943316 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:58.943551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:58.943511 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.044800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.044766 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.044975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.044937 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx5v8\" (UniqueName: \"kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.045034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.044990 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.045907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.045874 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.047169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.047147 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.069539 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.069510 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx5v8\" (UniqueName: \"kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8\") pod \"maas-api-revoke-sub-af8601f6-h97hl\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.175424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.175345 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:10:59.486297 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:10:59.486248 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl"] Oct 30 06:10:59.486885 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:10:59.486859 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod060da714_74a7_48a0_8298_690d068f558a.slice/crio-fec0ec760e9f975030d9051e70eede82209fa149c3816de114bae521ab7010af WatchSource:0}: Error finding container fec0ec760e9f975030d9051e70eede82209fa149c3816de114bae521ab7010af: Status 404 returned error can't find the container with id fec0ec760e9f975030d9051e70eede82209fa149c3816de114bae521ab7010af Oct 30 06:11:00.434956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:00.434912 5565 generic.go:356] "Generic (PLEG): container finished" podID="060da714-74a7-48a0-8298-690d068f558a" containerID="cbc0defa9827664b7a8030edddbbd53e892a0d3bb3292f5900aac1c85902f0d8" exitCode=0 Oct 30 06:11:00.435429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:00.434981 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" event={"ID":"060da714-74a7-48a0-8298-690d068f558a","Type":"ContainerDied","Data":"cbc0defa9827664b7a8030edddbbd53e892a0d3bb3292f5900aac1c85902f0d8"} Oct 30 06:11:00.435429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:00.435051 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" event={"ID":"060da714-74a7-48a0-8298-690d068f558a","Type":"ContainerStarted","Data":"fec0ec760e9f975030d9051e70eede82209fa149c3816de114bae521ab7010af"} Oct 30 06:11:01.610713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.610684 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:11:01.781614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.781566 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8\" (UniqueName: \"kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8\") pod \"060da714-74a7-48a0-8298-690d068f558a\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " Oct 30 06:11:01.781798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.781636 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca\") pod \"060da714-74a7-48a0-8298-690d068f558a\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " Oct 30 06:11:01.781798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.781698 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca\") pod \"060da714-74a7-48a0-8298-690d068f558a\" (UID: \"060da714-74a7-48a0-8298-690d068f558a\") " Oct 30 06:11:01.782084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.782060 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca" pod "060da714-74a7-48a0-8298-690d068f558a" (UID: "060da714-74a7-48a0-8298-690d068f558a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:01.782325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.782310 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/060da714-74a7-48a0-8298-690d068f558a-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.783840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.783815 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca" pod "060da714-74a7-48a0-8298-690d068f558a" (UID: "060da714-74a7-48a0-8298-690d068f558a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:01.783993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.783973 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8" pod "060da714-74a7-48a0-8298-690d068f558a" (UID: "060da714-74a7-48a0-8298-690d068f558a"). InnerVolumeSpecName "kube-api-access-wx5v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:01.883321 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.883282 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx5v8\" (UniqueName: \"kubernetes.io/projected/060da714-74a7-48a0-8298-690d068f558a-kube-api-access-wx5v8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:01.883321 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:01.883318 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/060da714-74a7-48a0-8298-690d068f558a-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:02.451683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:02.451649 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" Oct 30 06:11:02.451683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:02.451658 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl" event={"ID":"060da714-74a7-48a0-8298-690d068f558a","Type":"ContainerDied","Data":"fec0ec760e9f975030d9051e70eede82209fa149c3816de114bae521ab7010af"} Oct 30 06:11:02.451936 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:02.451708 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fec0ec760e9f975030d9051e70eede82209fa149c3816de114bae521ab7010af" Oct 30 06:11:09.341578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.341539 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9"] Oct 30 06:11:09.342511 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.342487 5565 cpu_manager.go:414] "containerMap: removing container" podUID="060da714-74a7-48a0-8298-690d068f558a" containerName="revoke-keys" Oct 30 06:11:09.342511 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.342511 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="060da714-74a7-48a0-8298-690d068f558a" containerName="revoke-keys" Oct 30 06:11:09.342780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.342764 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="060da714-74a7-48a0-8298-690d068f558a" containerName="revoke-keys" Oct 30 06:11:09.346855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.346831 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.366364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.366326 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9"] Oct 30 06:11:09.375892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.375857 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:09.376116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.375909 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:11:09.401269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.401226 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.401671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.401645 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766fw\" (UniqueName: \"kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.401910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.401893 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.503728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.503686 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.503973 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.503791 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766fw\" (UniqueName: \"kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.503973 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.503888 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.504508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.504478 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.506403 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.506379 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.529892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.529853 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766fw\" (UniqueName: \"kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw\") pod \"maas-api-revoke-sub-a6a98ec1-8qkf9\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.662030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.661934 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:09.980128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:09.980094 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9"] Oct 30 06:11:09.981068 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:11:09.981042 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04b1717a_35ca_47a6_99fc_ea41b86cb336.slice/crio-6b1112526b64636549bd8f1decd5ed0c781c604d841b447e088ce9cd54f91552 WatchSource:0}: Error finding container 6b1112526b64636549bd8f1decd5ed0c781c604d841b447e088ce9cd54f91552: Status 404 returned error can't find the container with id 6b1112526b64636549bd8f1decd5ed0c781c604d841b447e088ce9cd54f91552 Oct 30 06:11:10.530356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:10.530263 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" event={"ID":"04b1717a-35ca-47a6-99fc-ea41b86cb336","Type":"ContainerStarted","Data":"aca1205b8518e597178798885ed218082f837251b69104dd5dd76db0cb8b043c"} Oct 30 06:11:10.530356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:10.530319 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" event={"ID":"04b1717a-35ca-47a6-99fc-ea41b86cb336","Type":"ContainerStarted","Data":"6b1112526b64636549bd8f1decd5ed0c781c604d841b447e088ce9cd54f91552"} Oct 30 06:11:10.576223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:10.576172 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" podStartSLOduration=1.576158928 podStartE2EDuration="1.576158928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:10.573775117 +0000 UTC m=+2377.149462173" watchObservedRunningTime="2026-10-30 06:11:10.576158928 +0000 UTC m=+2377.151845982" Oct 30 06:11:11.539548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:11.539505 5565 generic.go:356] "Generic (PLEG): container finished" podID="04b1717a-35ca-47a6-99fc-ea41b86cb336" containerID="aca1205b8518e597178798885ed218082f837251b69104dd5dd76db0cb8b043c" exitCode=0 Oct 30 06:11:11.539950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:11.539570 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" event={"ID":"04b1717a-35ca-47a6-99fc-ea41b86cb336","Type":"ContainerDied","Data":"aca1205b8518e597178798885ed218082f837251b69104dd5dd76db0cb8b043c"} Oct 30 06:11:12.719894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.719865 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:12.853960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.853881 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca\") pod \"04b1717a-35ca-47a6-99fc-ea41b86cb336\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " Oct 30 06:11:12.854137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.853983 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca\") pod \"04b1717a-35ca-47a6-99fc-ea41b86cb336\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " Oct 30 06:11:12.854137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.854051 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw\" (UniqueName: \"kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw\") pod \"04b1717a-35ca-47a6-99fc-ea41b86cb336\" (UID: \"04b1717a-35ca-47a6-99fc-ea41b86cb336\") " Oct 30 06:11:12.854326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.854302 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca" pod "04b1717a-35ca-47a6-99fc-ea41b86cb336" (UID: "04b1717a-35ca-47a6-99fc-ea41b86cb336"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:12.854449 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.854435 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:12.856096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.856071 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca" pod "04b1717a-35ca-47a6-99fc-ea41b86cb336" (UID: "04b1717a-35ca-47a6-99fc-ea41b86cb336"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:12.856193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.856114 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw" pod "04b1717a-35ca-47a6-99fc-ea41b86cb336" (UID: "04b1717a-35ca-47a6-99fc-ea41b86cb336"). InnerVolumeSpecName "kube-api-access-766fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:12.956160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.956133 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/04b1717a-35ca-47a6-99fc-ea41b86cb336-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:12.956160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:12.956162 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766fw\" (UniqueName: \"kubernetes.io/projected/04b1717a-35ca-47a6-99fc-ea41b86cb336-kube-api-access-766fw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:13.555727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:13.555695 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" Oct 30 06:11:13.555904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:13.555721 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9" event={"ID":"04b1717a-35ca-47a6-99fc-ea41b86cb336","Type":"ContainerDied","Data":"6b1112526b64636549bd8f1decd5ed0c781c604d841b447e088ce9cd54f91552"} Oct 30 06:11:13.555904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:13.555783 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b1112526b64636549bd8f1decd5ed0c781c604d841b447e088ce9cd54f91552" Oct 30 06:11:17.535623 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:11:17.535576 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:11:22.096396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:22.096356 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg"] Oct 30 06:11:22.096971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:22.096600 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" containerID="cri-o://649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37" gracePeriod=30 Oct 30 06:11:22.753557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:22.745132 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:11:26.033504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:26.033461 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p"] Oct 30 06:11:26.047170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:26.047136 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b942db37-r5z4p"] Oct 30 06:11:26.741782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:26.741748 5565 patch_prober.go:29] interesting pod/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:11:26.741976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:26.741814 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:11:27.415616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.415591 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 06:11:27.433645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433621 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.433774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433655 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.433774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433675 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.433774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433727 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.433958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433846 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.433958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433898 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:27.433958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.433925 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.434126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434030 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.434126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434078 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:27.434245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434152 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.434245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434193 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket\") pod \"de36eef2-9a73-4600-848e-650ff0c1612f\" (UID: \"de36eef2-9a73-4600-848e-650ff0c1612f\") " Oct 30 06:11:27.434363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434288 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:27.434459 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434416 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:27.434693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434667 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:27.434879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434859 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.434954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434883 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.434954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434898 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.434954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434912 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.434954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.434927 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de36eef2-9a73-4600-848e-650ff0c1612f-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.436161 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.436141 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:11:27.436243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.436180 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "kube-api-access-zg8wr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:27.436448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.436430 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:11:27.436515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.436440 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token" pod "de36eef2-9a73-4600-848e-650ff0c1612f" (UID: "de36eef2-9a73-4600-848e-650ff0c1612f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:27.536663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.536633 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de36eef2-9a73-4600-848e-650ff0c1612f-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.536663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.536659 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.536663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.536669 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg8wr\" (UniqueName: \"kubernetes.io/projected/de36eef2-9a73-4600-848e-650ff0c1612f-kube-api-access-zg8wr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.536896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.536679 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de36eef2-9a73-4600-848e-650ff0c1612f-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:27.569169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.569139 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c959a373-cca2-4f23-bac1-05b1c4eba284" path="/var/lib/kubelet/pods/c959a373-cca2-4f23-bac1-05b1c4eba284/volumes" Oct 30 06:11:27.673795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.673702 5565 generic.go:356] "Generic (PLEG): container finished" podID="de36eef2-9a73-4600-848e-650ff0c1612f" containerID="649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37" exitCode=0 Oct 30 06:11:27.673795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.673789 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" event={"ID":"de36eef2-9a73-4600-848e-650ff0c1612f","Type":"ContainerDied","Data":"649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37"} Oct 30 06:11:27.674054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.673868 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" event={"ID":"de36eef2-9a73-4600-848e-650ff0c1612f","Type":"ContainerDied","Data":"54f29f1c7cb4c0a127c30ebcb65f8b114a28950ce9b66c4af7fd8cd8b7c71433"} Oct 30 06:11:27.674054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.673909 5565 scope.go:122] "RemoveContainer" containerID="649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37" Oct 30 06:11:27.674169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.674130 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg" Oct 30 06:11:27.686884 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.686856 5565 scope.go:122] "RemoveContainer" containerID="649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37" Oct 30 06:11:27.687320 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:11:27.687274 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37\": container with ID starting with 649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37 not found: ID does not exist" containerID="649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37" Oct 30 06:11:27.687392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.687340 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37"} err="failed to get container status \"649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37\": rpc error: code = NotFound desc = could not find container \"649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37\": container with ID starting with 649674debb5e4cbeac080a29a052aa961de3ebe5e64ea67cf55012bf9bcf4c37 not found: ID does not exist" Oct 30 06:11:27.734212 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.734183 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg"] Oct 30 06:11:27.738099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:27.738075 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-s47hg"] Oct 30 06:11:29.570445 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:29.570414 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" path="/var/lib/kubelet/pods/de36eef2-9a73-4600-848e-650ff0c1612f/volumes" Oct 30 06:11:30.701547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.701500 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf"] Oct 30 06:11:30.702283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.702265 5565 cpu_manager.go:414] "containerMap: removing container" podUID="04b1717a-35ca-47a6-99fc-ea41b86cb336" containerName="revoke-keys" Oct 30 06:11:30.702332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.702284 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04b1717a-35ca-47a6-99fc-ea41b86cb336" containerName="revoke-keys" Oct 30 06:11:30.702332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.702304 5565 cpu_manager.go:414] "containerMap: removing container" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" Oct 30 06:11:30.702332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.702309 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" Oct 30 06:11:30.702436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.702425 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="04b1717a-35ca-47a6-99fc-ea41b86cb336" containerName="revoke-keys" Oct 30 06:11:30.702470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.702436 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="de36eef2-9a73-4600-848e-650ff0c1612f" containerName="istio-proxy" Oct 30 06:11:30.706045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.706029 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.723950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.723925 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf"] Oct 30 06:11:30.733615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.733588 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:30.734229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.734209 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:11:30.783906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.783878 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffszd\" (UniqueName: \"kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.784045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.783941 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.784148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.784123 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.885896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.885870 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.886024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.885991 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffszd\" (UniqueName: \"kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.886067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.886037 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.886492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.886467 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.888448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.888425 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:30.911003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:30.910975 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffszd\" (UniqueName: \"kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd\") pod \"maas-api-revoke-sub-edfa701c-pssmf\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:31.021519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.021483 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:31.331404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.331378 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf"] Oct 30 06:11:31.334271 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:11:31.334240 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca947cce_3800_4ed7_818f_5495b308430d.slice/crio-228c9178fd241876d62c1441904d51df6a8558f3b3174ddd5fb8b44a8fa72fd1 WatchSource:0}: Error finding container 228c9178fd241876d62c1441904d51df6a8558f3b3174ddd5fb8b44a8fa72fd1: Status 404 returned error can't find the container with id 228c9178fd241876d62c1441904d51df6a8558f3b3174ddd5fb8b44a8fa72fd1 Oct 30 06:11:31.601097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.601028 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq"] Oct 30 06:11:31.606610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.606590 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.636941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.636910 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq"] Oct 30 06:11:31.702083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.702053 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.702462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.702230 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.702462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.702261 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4fqz\" (UniqueName: \"kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.707237 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.707205 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" event={"ID":"ca947cce-3800-4ed7-818f-5495b308430d","Type":"ContainerStarted","Data":"f9abfaabc61c77793bc982da96d25587ae8ff0fd79567399f3311d34531b38d4"} Oct 30 06:11:31.707347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.707262 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" event={"ID":"ca947cce-3800-4ed7-818f-5495b308430d","Type":"ContainerStarted","Data":"228c9178fd241876d62c1441904d51df6a8558f3b3174ddd5fb8b44a8fa72fd1"} Oct 30 06:11:31.753287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.753242 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" podStartSLOduration=1.753228364 podStartE2EDuration="1.753228364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:31.751028646 +0000 UTC m=+2398.326715700" watchObservedRunningTime="2026-10-30 06:11:31.753228364 +0000 UTC m=+2398.328915418" Oct 30 06:11:31.803683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.803640 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.803871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.803808 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.803871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.803836 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4fqz\" (UniqueName: \"kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.804487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.804459 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.806054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.806024 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.828974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.828945 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4fqz\" (UniqueName: \"kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz\") pod \"maas-api-revoke-sub-7b1f55a5-s2wmq\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:31.931866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:31.931789 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:32.237258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:32.237227 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq"] Oct 30 06:11:32.238262 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:11:32.238234 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod864ed9ac_7d96_4c1b_86d5_c1c27bfa17af.slice/crio-91ba1188160491bf0b7239eed2cbf022c83bbff1b99896424668d34ece75835e WatchSource:0}: Error finding container 91ba1188160491bf0b7239eed2cbf022c83bbff1b99896424668d34ece75835e: Status 404 returned error can't find the container with id 91ba1188160491bf0b7239eed2cbf022c83bbff1b99896424668d34ece75835e Oct 30 06:11:32.716926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:32.716890 5565 generic.go:356] "Generic (PLEG): container finished" podID="ca947cce-3800-4ed7-818f-5495b308430d" containerID="f9abfaabc61c77793bc982da96d25587ae8ff0fd79567399f3311d34531b38d4" exitCode=0 Oct 30 06:11:32.717342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:32.716976 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" event={"ID":"ca947cce-3800-4ed7-818f-5495b308430d","Type":"ContainerDied","Data":"f9abfaabc61c77793bc982da96d25587ae8ff0fd79567399f3311d34531b38d4"} Oct 30 06:11:32.718936 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:32.718912 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" event={"ID":"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af","Type":"ContainerStarted","Data":"f2e17eb72f238db4c0d93a9b53f88f5a84c1cf3d488463bd58a7653eea41b870"} Oct 30 06:11:32.719092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:32.718955 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" event={"ID":"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af","Type":"ContainerStarted","Data":"91ba1188160491bf0b7239eed2cbf022c83bbff1b99896424668d34ece75835e"} Oct 30 06:11:32.813116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:32.813051 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" podStartSLOduration=1.813031782 podStartE2EDuration="1.813031782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:11:32.811981419 +0000 UTC m=+2399.387668502" watchObservedRunningTime="2026-10-30 06:11:32.813031782 +0000 UTC m=+2399.388718837" Oct 30 06:11:33.727675 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:33.727589 5565 generic.go:356] "Generic (PLEG): container finished" podID="864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" containerID="f2e17eb72f238db4c0d93a9b53f88f5a84c1cf3d488463bd58a7653eea41b870" exitCode=0 Oct 30 06:11:33.727675 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:33.727663 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" event={"ID":"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af","Type":"ContainerDied","Data":"f2e17eb72f238db4c0d93a9b53f88f5a84c1cf3d488463bd58a7653eea41b870"} Oct 30 06:11:33.927133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:33.927106 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:34.060989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.060961 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd\" (UniqueName: \"kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd\") pod \"ca947cce-3800-4ed7-818f-5495b308430d\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " Oct 30 06:11:34.061160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.061106 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca\") pod \"ca947cce-3800-4ed7-818f-5495b308430d\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " Oct 30 06:11:34.061222 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.061172 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca\") pod \"ca947cce-3800-4ed7-818f-5495b308430d\" (UID: \"ca947cce-3800-4ed7-818f-5495b308430d\") " Oct 30 06:11:34.061566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.061510 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca" pod "ca947cce-3800-4ed7-818f-5495b308430d" (UID: "ca947cce-3800-4ed7-818f-5495b308430d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:34.061830 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.061810 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:34.063235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.063211 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd" pod "ca947cce-3800-4ed7-818f-5495b308430d" (UID: "ca947cce-3800-4ed7-818f-5495b308430d"). InnerVolumeSpecName "kube-api-access-ffszd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:34.063336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.063227 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca" pod "ca947cce-3800-4ed7-818f-5495b308430d" (UID: "ca947cce-3800-4ed7-818f-5495b308430d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:34.162944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.162911 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca947cce-3800-4ed7-818f-5495b308430d-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:34.162944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.162944 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffszd\" (UniqueName: \"kubernetes.io/projected/ca947cce-3800-4ed7-818f-5495b308430d-kube-api-access-ffszd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:34.737398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.737360 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" Oct 30 06:11:34.737398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.737376 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf" event={"ID":"ca947cce-3800-4ed7-818f-5495b308430d","Type":"ContainerDied","Data":"228c9178fd241876d62c1441904d51df6a8558f3b3174ddd5fb8b44a8fa72fd1"} Oct 30 06:11:34.737885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.737430 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="228c9178fd241876d62c1441904d51df6a8558f3b3174ddd5fb8b44a8fa72fd1" Oct 30 06:11:34.888746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.888718 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:34.973792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.973758 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca\") pod \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " Oct 30 06:11:34.974048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.974028 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz\" (UniqueName: \"kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz\") pod \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " Oct 30 06:11:34.974127 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.974080 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca\") pod \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\" (UID: \"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af\") " Oct 30 06:11:34.974472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.974442 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca" pod "864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" (UID: "864ed9ac-7d96-4c1b-86d5-c1c27bfa17af"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:34.974841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.974821 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:34.976009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.975986 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca" pod "864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" (UID: "864ed9ac-7d96-4c1b-86d5-c1c27bfa17af"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:34.976640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:34.976622 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz" pod "864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" (UID: "864ed9ac-7d96-4c1b-86d5-c1c27bfa17af"). InnerVolumeSpecName "kube-api-access-v4fqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:35.075792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:35.075757 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4fqz\" (UniqueName: \"kubernetes.io/projected/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-kube-api-access-v4fqz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:35.075792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:35.075788 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:35.747673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:35.747639 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" event={"ID":"864ed9ac-7d96-4c1b-86d5-c1c27bfa17af","Type":"ContainerDied","Data":"91ba1188160491bf0b7239eed2cbf022c83bbff1b99896424668d34ece75835e"} Oct 30 06:11:35.747673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:35.747664 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq" Oct 30 06:11:35.748104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:35.747689 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91ba1188160491bf0b7239eed2cbf022c83bbff1b99896424668d34ece75835e" Oct 30 06:11:41.097498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.097459 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4"] Oct 30 06:11:41.098200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.098179 5565 cpu_manager.go:414] "containerMap: removing container" podUID="864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" containerName="revoke-keys" Oct 30 06:11:41.098200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.098200 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" containerName="revoke-keys" Oct 30 06:11:41.098313 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.098234 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ca947cce-3800-4ed7-818f-5495b308430d" containerName="revoke-keys" Oct 30 06:11:41.098313 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.098241 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca947cce-3800-4ed7-818f-5495b308430d" containerName="revoke-keys" Oct 30 06:11:41.098388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.098377 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca947cce-3800-4ed7-818f-5495b308430d" containerName="revoke-keys" Oct 30 06:11:41.098424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.098397 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" containerName="revoke-keys" Oct 30 06:11:41.103642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.103618 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.122392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.122365 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4"] Oct 30 06:11:41.136561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.131192 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:11:41.136561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.131495 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:11:41.272551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.271866 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.272551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.272056 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4dlh\" (UniqueName: \"kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.272551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.272321 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.374139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.374049 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.374139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.374126 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4dlh\" (UniqueName: \"kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.374329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.374275 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.374782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.374760 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.376719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.376697 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.399513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.399482 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4dlh\" (UniqueName: \"kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh\") pod \"maas-api-revoke-sub-c5663160-snfr4\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.417986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.417953 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:41.739583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.739421 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4"] Oct 30 06:11:41.751433 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:11:41.751399 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda48e662a_4211_4b5c_bb5c_3ee182ef7175.slice/crio-cb24ab398277fe2981f20919e1458da3e8ea47e52e6cef79a41cd5ef23b48a16 WatchSource:0}: Error finding container cb24ab398277fe2981f20919e1458da3e8ea47e52e6cef79a41cd5ef23b48a16: Status 404 returned error can't find the container with id cb24ab398277fe2981f20919e1458da3e8ea47e52e6cef79a41cd5ef23b48a16 Oct 30 06:11:41.801200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:41.801165 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" event={"ID":"a48e662a-4211-4b5c-bb5c-3ee182ef7175","Type":"ContainerStarted","Data":"cb24ab398277fe2981f20919e1458da3e8ea47e52e6cef79a41cd5ef23b48a16"} Oct 30 06:11:42.811333 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:42.811296 5565 generic.go:356] "Generic (PLEG): container finished" podID="a48e662a-4211-4b5c-bb5c-3ee182ef7175" containerID="f062d8e88c1ac5b35c267974064cfeb20ec76dff0b3a2f675b54fa38b9d0b1f9" exitCode=0 Oct 30 06:11:42.811803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:42.811339 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" event={"ID":"a48e662a-4211-4b5c-bb5c-3ee182ef7175","Type":"ContainerDied","Data":"f062d8e88c1ac5b35c267974064cfeb20ec76dff0b3a2f675b54fa38b9d0b1f9"} Oct 30 06:11:43.995599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:43.995563 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:44.119441 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.119176 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca\") pod \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " Oct 30 06:11:44.121152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.120742 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh\" (UniqueName: \"kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh\") pod \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " Oct 30 06:11:44.121714 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.121668 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca\") pod \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\" (UID: \"a48e662a-4211-4b5c-bb5c-3ee182ef7175\") " Oct 30 06:11:44.122457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.122423 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca" pod "a48e662a-4211-4b5c-bb5c-3ee182ef7175" (UID: "a48e662a-4211-4b5c-bb5c-3ee182ef7175"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:11:44.122736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.122705 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:44.124746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.124718 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh" pod "a48e662a-4211-4b5c-bb5c-3ee182ef7175" (UID: "a48e662a-4211-4b5c-bb5c-3ee182ef7175"). InnerVolumeSpecName "kube-api-access-k4dlh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:11:44.124894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.124742 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca" pod "a48e662a-4211-4b5c-bb5c-3ee182ef7175" (UID: "a48e662a-4211-4b5c-bb5c-3ee182ef7175"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:11:44.224053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.224017 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a48e662a-4211-4b5c-bb5c-3ee182ef7175-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:44.224053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.224049 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k4dlh\" (UniqueName: \"kubernetes.io/projected/a48e662a-4211-4b5c-bb5c-3ee182ef7175-kube-api-access-k4dlh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:11:44.828499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.828467 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" Oct 30 06:11:44.828747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.828494 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4" event={"ID":"a48e662a-4211-4b5c-bb5c-3ee182ef7175","Type":"ContainerDied","Data":"cb24ab398277fe2981f20919e1458da3e8ea47e52e6cef79a41cd5ef23b48a16"} Oct 30 06:11:44.828747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:44.828584 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb24ab398277fe2981f20919e1458da3e8ea47e52e6cef79a41cd5ef23b48a16" Oct 30 06:11:47.536807 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:11:47.536754 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:11:57.038801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:57.038761 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8"] Oct 30 06:11:57.051196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:57.051163 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01269541-tjpd8"] Oct 30 06:11:57.568484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:11:57.568455 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e477b94-9bb7-4437-bc29-ebea4fbdd33b" path="/var/lib/kubelet/pods/0e477b94-9bb7-4437-bc29-ebea4fbdd33b/volumes" Oct 30 06:12:00.224980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.224950 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp"] Oct 30 06:12:00.225661 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.225645 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a48e662a-4211-4b5c-bb5c-3ee182ef7175" containerName="revoke-keys" Oct 30 06:12:00.225718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.225662 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a48e662a-4211-4b5c-bb5c-3ee182ef7175" containerName="revoke-keys" Oct 30 06:12:00.225811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.225802 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a48e662a-4211-4b5c-bb5c-3ee182ef7175" containerName="revoke-keys" Oct 30 06:12:00.230821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.230803 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.257392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.257367 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-c201cd-serving-cert\"" Oct 30 06:12:00.276013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.275984 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp"] Oct 30 06:12:00.288367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.288312 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.288518 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.288418 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lv7f\" (UniqueName: \"kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.288518 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.288467 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.288518 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.288500 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.389844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.389809 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lv7f\" (UniqueName: \"kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.390045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.389855 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.390045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.389874 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.390045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.390037 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.390621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.390598 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.390928 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:12:00.390908 5565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-serving-cert: secret "maas-api-e2e-ipp-a-c201cd-serving-cert" not found Oct 30 06:12:00.390996 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:12:00.390986 5565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls podName:517508c8-aeb3-425a-8ea4-3720a23aee80 nodeName:}" failed. No retries permitted until 2026-10-30 06:12:00.89097037 +0000 UTC m=+2427.466657420 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls") pod "maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" (UID: "517508c8-aeb3-425a-8ea4-3720a23aee80") : secret "maas-api-e2e-ipp-a-c201cd-serving-cert" not found Oct 30 06:12:00.392386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.392366 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.415046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.415017 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lv7f\" (UniqueName: \"kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.897921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.897882 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:00.900475 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:00.900452 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\") pod \"maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:01.146840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:01.146792 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:01.488012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:01.487973 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp"] Oct 30 06:12:01.488400 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:12:01.488088 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod517508c8_aeb3_425a_8ea4_3720a23aee80.slice/crio-e15c774d17ac9ac6e724f4cce203447e786f9cba24b57757ad75ff5939fc115f WatchSource:0}: Error finding container e15c774d17ac9ac6e724f4cce203447e786f9cba24b57757ad75ff5939fc115f: Status 404 returned error can't find the container with id e15c774d17ac9ac6e724f4cce203447e786f9cba24b57757ad75ff5939fc115f Oct 30 06:12:01.489271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:01.489252 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:12:02.000104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:02.000059 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" event={"ID":"517508c8-aeb3-425a-8ea4-3720a23aee80","Type":"ContainerStarted","Data":"6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5"} Oct 30 06:12:02.000287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:02.000122 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" event={"ID":"517508c8-aeb3-425a-8ea4-3720a23aee80","Type":"ContainerStarted","Data":"e15c774d17ac9ac6e724f4cce203447e786f9cba24b57757ad75ff5939fc115f"} Oct 30 06:12:02.000362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:02.000342 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:02.048879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:02.048831 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" podStartSLOduration=1.691827068 podStartE2EDuration="2.048818121s" totalImagesPullingTime="356.991053ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:02.045929256 +0000 UTC m=+2428.621616310" watchObservedRunningTime="2026-10-30 06:12:02.048818121 +0000 UTC m=+2428.624505176" Oct 30 06:12:08.013287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.013258 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:12:08.032167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.032141 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c"] Oct 30 06:12:08.047816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.047791 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f51265a6-tzn2c"] Oct 30 06:12:08.518711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.518679 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz"] Oct 30 06:12:08.523488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.523470 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.555727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.555700 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz"] Oct 30 06:12:08.613905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.613875 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.614076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.614019 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.614076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.614062 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.614289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.614265 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4grc\" (UniqueName: \"kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.631960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.631923 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w6-95498c-serving-cert\"" Oct 30 06:12:08.715491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.715461 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.715643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.715542 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.715643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.715626 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4grc\" (UniqueName: \"kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.715803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.715787 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.716236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.716215 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.718020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.717998 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.718124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.718043 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.741095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.741066 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4grc\" (UniqueName: \"kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc\") pod \"maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:08.839410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:08.839315 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:09.158700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:09.158674 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz"] Oct 30 06:12:09.159073 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:12:09.158983 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9412d483_a3c7_4bb4_89bf_d620bb74c382.slice/crio-9fd73a66e0027ed6283f5cd6f099230e3c483aa6eeca3102d0fc0274686ee6ee WatchSource:0}: Error finding container 9fd73a66e0027ed6283f5cd6f099230e3c483aa6eeca3102d0fc0274686ee6ee: Status 404 returned error can't find the container with id 9fd73a66e0027ed6283f5cd6f099230e3c483aa6eeca3102d0fc0274686ee6ee Oct 30 06:12:09.568618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:09.568588 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="863e15bc-6719-4783-970d-a757bddbe9ec" path="/var/lib/kubelet/pods/863e15bc-6719-4783-970d-a757bddbe9ec/volumes" Oct 30 06:12:10.064167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:10.064132 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" event={"ID":"9412d483-a3c7-4bb4-89bf-d620bb74c382","Type":"ContainerStarted","Data":"f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b"} Oct 30 06:12:10.064354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:10.064181 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" event={"ID":"9412d483-a3c7-4bb4-89bf-d620bb74c382","Type":"ContainerStarted","Data":"9fd73a66e0027ed6283f5cd6f099230e3c483aa6eeca3102d0fc0274686ee6ee"} Oct 30 06:12:10.064354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:10.064333 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:10.124203 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:10.124151 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" podStartSLOduration=1.8309642959999999 podStartE2EDuration="2.124139447s" totalImagesPullingTime="293.175151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:10.122988324 +0000 UTC m=+2436.698675388" watchObservedRunningTime="2026-10-30 06:12:10.124139447 +0000 UTC m=+2436.699826502" Oct 30 06:12:13.878174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:13.878131 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:12:16.079062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:16.079029 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:12:17.535233 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:12:17.535190 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:12:21.123282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.123246 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp"] Oct 30 06:12:21.129812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.129792 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.149766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.149733 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp"] Oct 30 06:12:21.157931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.157905 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36ae27\"" Oct 30 06:12:21.179411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.179378 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4qmk\" (UniqueName: \"kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.179669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.179646 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.281206 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.281172 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4qmk\" (UniqueName: \"kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.281386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.281327 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.282056 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.282029 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.358328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.358287 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4qmk\" (UniqueName: \"kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.446750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.446053 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:21.793509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:21.793472 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp"] Oct 30 06:12:21.793740 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:12:21.793717 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod313637b4_a045_4e66_9064_7c59b12a1da2.slice/crio-0a14a533c29eeb5190c2d17b536bae0dcfc3ca4bb0813ab69c47489c90759ee2 WatchSource:0}: Error finding container 0a14a533c29eeb5190c2d17b536bae0dcfc3ca4bb0813ab69c47489c90759ee2: Status 404 returned error can't find the container with id 0a14a533c29eeb5190c2d17b536bae0dcfc3ca4bb0813ab69c47489c90759ee2 Oct 30 06:12:22.244584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:22.244481 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" event={"ID":"313637b4-a045-4e66-9064-7c59b12a1da2","Type":"ContainerStarted","Data":"0e25ab6f1f4fc005b7f393f047fae0c120a200fabcf29b37da1384451bce4bc5"} Oct 30 06:12:22.244584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:22.244568 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" event={"ID":"313637b4-a045-4e66-9064-7c59b12a1da2","Type":"ContainerStarted","Data":"0a14a533c29eeb5190c2d17b536bae0dcfc3ca4bb0813ab69c47489c90759ee2"} Oct 30 06:12:22.304612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:22.297250 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" podStartSLOduration=1.2972147920000001 podStartE2EDuration="1.297214792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:22.288850854 +0000 UTC m=+2448.864537909" watchObservedRunningTime="2026-10-30 06:12:22.297214792 +0000 UTC m=+2448.872901848" Oct 30 06:12:23.252147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:23.252120 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:26.297658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.297617 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw"] Oct 30 06:12:26.302519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.302497 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.326004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.325977 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw"] Oct 30 06:12:26.334141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.334102 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ebe61e\"" Oct 30 06:12:26.372829 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.372795 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5hg6\" (UniqueName: \"kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.372987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.372943 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.478597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.478551 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.500484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.483879 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5hg6\" (UniqueName: \"kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.500484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.479736 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.519219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.519187 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5hg6\" (UniqueName: \"kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.617823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.617732 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:26.928017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:26.927977 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw"] Oct 30 06:12:26.928449 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:12:26.928422 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08863796_4736_4be3_8d67_6bf865f69983.slice/crio-28a1240647f16dd87182ccf890e4156ffb1e3746257538ef4559ced92e65952a WatchSource:0}: Error finding container 28a1240647f16dd87182ccf890e4156ffb1e3746257538ef4559ced92e65952a: Status 404 returned error can't find the container with id 28a1240647f16dd87182ccf890e4156ffb1e3746257538ef4559ced92e65952a Oct 30 06:12:27.283948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:27.283914 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" event={"ID":"08863796-4736-4be3-8d67-6bf865f69983","Type":"ContainerStarted","Data":"75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1"} Oct 30 06:12:27.284438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:27.283964 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" event={"ID":"08863796-4736-4be3-8d67-6bf865f69983","Type":"ContainerStarted","Data":"28a1240647f16dd87182ccf890e4156ffb1e3746257538ef4559ced92e65952a"} Oct 30 06:12:27.284438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:27.284032 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:27.336732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:27.336671 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" podStartSLOduration=1.336656219 podStartE2EDuration="1.336656219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:27.334759912 +0000 UTC m=+2453.910446967" watchObservedRunningTime="2026-10-30 06:12:27.336656219 +0000 UTC m=+2453.912343273" Oct 30 06:12:28.439452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.439422 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn"] Oct 30 06:12:28.443868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.443846 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.477648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.477618 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn"] Oct 30 06:12:28.489762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.489627 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-6b8f87-openshift-default-dockercfg-pn4kl\"" Oct 30 06:12:28.513401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.513365 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.513672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.513647 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.513876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.513858 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.514054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.514037 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.514270 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.514250 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.514413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.514398 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.514573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.514555 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8br5k\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.514854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.514834 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.515006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.514991 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.616624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616590 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.616824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616677 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.616824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616703 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.616824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616726 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8br5k\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616873 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616902 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616939 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.616973 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.617036 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.617252 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.617382 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617474 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.617440 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.617490 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.617846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.617820 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.619438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.619410 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.619780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.619761 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.644382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.644351 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.645537 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.645506 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8br5k\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k\") pod \"e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:28.789423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:28.789377 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:29.122971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.122941 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn"] Oct 30 06:12:29.125100 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:12:29.125074 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61891b18_0cc3_42d4_9866_8da70e8e06e1.slice/crio-947201b8631729064c5198cf547b4a221dcb5d3e4c01ee5ebbf30b8b5d25174d WatchSource:0}: Error finding container 947201b8631729064c5198cf547b4a221dcb5d3e4c01ee5ebbf30b8b5d25174d: Status 404 returned error can't find the container with id 947201b8631729064c5198cf547b4a221dcb5d3e4c01ee5ebbf30b8b5d25174d Oct 30 06:12:29.127912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.127880 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:12:29.128001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.127945 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:12:29.128001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.127977 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:12:29.302686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.302645 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" event={"ID":"61891b18-0cc3-42d4-9866-8da70e8e06e1","Type":"ContainerStarted","Data":"434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759"} Oct 30 06:12:29.302868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.302714 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" event={"ID":"61891b18-0cc3-42d4-9866-8da70e8e06e1","Type":"ContainerStarted","Data":"947201b8631729064c5198cf547b4a221dcb5d3e4c01ee5ebbf30b8b5d25174d"} Oct 30 06:12:29.368933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.368878 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podStartSLOduration=1.368862226 podStartE2EDuration="1.368862226s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:12:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:12:29.359438387 +0000 UTC m=+2455.935125443" watchObservedRunningTime="2026-10-30 06:12:29.368862226 +0000 UTC m=+2455.944549329" Oct 30 06:12:29.789796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:29.789762 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:30.791907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:30.791866 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 06:12:30.792441 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:30.791934 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 06:12:31.790186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:31.790152 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 06:12:31.790380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:31.790207 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 06:12:32.790555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:32.790485 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 06:12:32.791052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:32.790591 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 06:12:33.790737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:33.790704 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 06:12:33.791152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:33.790756 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 06:12:34.281006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:34.280967 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:12:34.790675 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:34.790638 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 06:12:34.790861 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:34.790693 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 06:12:35.789776 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:35.789733 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" start-of-body= Oct 30 06:12:35.790001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:35.789788 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.234:15021/healthz/ready\": dial tcp 10.134.0.234:15021: connect: connection refused" Oct 30 06:12:36.795269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:36.795234 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:36.795699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:36.795685 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:36.796379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:36.796355 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:12:38.295600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:38.295568 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:12:38.328163 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:38.328130 5565 scope.go:122] "RemoveContainer" containerID="21946dddfdf15c993773ecef58f7c9611ce01b11a9a919c31be052804ab6ee5e" Oct 30 06:12:38.344761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:12:38.344732 5565 scope.go:122] "RemoveContainer" containerID="fd44dfdfe1ca50afd29043eb5f87f2c1cb880b26d6834a77d13b1a55fd9cec3e" Oct 30 06:12:47.533931 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:12:47.533889 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:13:00.388574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.388522 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b"] Oct 30 06:13:00.395102 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.395079 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.413117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.413089 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-36ae27-serving-cert\"" Oct 30 06:13:00.416314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.416286 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b"] Oct 30 06:13:00.510432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.510397 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c96b6\" (UniqueName: \"kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.510649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.510508 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.510712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.510692 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.510768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.510740 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.630609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.630562 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c96b6\" (UniqueName: \"kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.630821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.630682 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.630821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.630803 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.630948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.630847 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.632103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.632074 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.665599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.664601 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.674547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.674508 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c96b6\" (UniqueName: \"kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.689414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.689373 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls\") pod \"maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:00.712262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:00.712227 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:01.024915 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:13:01.024876 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9307b9d2_65e1_4c16_952c_817bf260daa4.slice/crio-2cf108339781aba4d77ebab15a6d048c1e3ba6a731f9dbf854f41f792f2eaf38 WatchSource:0}: Error finding container 2cf108339781aba4d77ebab15a6d048c1e3ba6a731f9dbf854f41f792f2eaf38: Status 404 returned error can't find the container with id 2cf108339781aba4d77ebab15a6d048c1e3ba6a731f9dbf854f41f792f2eaf38 Oct 30 06:13:01.025041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:01.024928 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b"] Oct 30 06:13:01.745279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:01.745176 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" event={"ID":"9307b9d2-65e1-4c16-952c-817bf260daa4","Type":"ContainerStarted","Data":"b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d"} Oct 30 06:13:01.745813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:01.745282 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:01.745813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:01.745303 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" event={"ID":"9307b9d2-65e1-4c16-952c-817bf260daa4","Type":"ContainerStarted","Data":"2cf108339781aba4d77ebab15a6d048c1e3ba6a731f9dbf854f41f792f2eaf38"} Oct 30 06:13:01.799504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:01.799434 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" podStartSLOduration=1.466588605 podStartE2EDuration="1.799413767s" totalImagesPullingTime="332.825162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:01.794846492 +0000 UTC m=+2488.370533547" watchObservedRunningTime="2026-10-30 06:13:01.799413767 +0000 UTC m=+2488.375100822" Oct 30 06:13:03.042886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:03.042850 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq"] Oct 30 06:13:03.066550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:03.063902 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-518c4c4e-g6czq"] Oct 30 06:13:03.628384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:03.628324 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac4d83ca-9be4-42e7-95b6-2e8110ec5943" path="/var/lib/kubelet/pods/ac4d83ca-9be4-42e7-95b6-2e8110ec5943/volumes" Oct 30 06:13:07.760171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:07.760140 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:13:17.533060 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:13:17.533025 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:13:31.281762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.281722 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b"] Oct 30 06:13:31.286232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.286213 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.303278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.303254 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-c201cd\"" Oct 30 06:13:31.319866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.319830 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-c201cd-dockercfg-vmn8x\"" Oct 30 06:13:31.346355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.336223 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b"] Oct 30 06:13:31.385765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.373733 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdw97\" (UniqueName: \"kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97\") pod \"payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.386110 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.386085 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.488736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.488695 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdw97\" (UniqueName: \"kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97\") pod \"payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.489071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.489041 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.491575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.491549 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.534293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.534037 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdw97\" (UniqueName: \"kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97\") pod \"payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.601084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.601053 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:31.913256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:31.913230 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b"] Oct 30 06:13:31.914324 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:13:31.914299 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2753399b_3bdc_4c2a_b41b_c941c1af7271.slice/crio-bf2f5277657e1cfd9b58f6407875bf2dcf8ba455b0f127510ce11493bcc9154e WatchSource:0}: Error finding container bf2f5277657e1cfd9b58f6407875bf2dcf8ba455b0f127510ce11493bcc9154e: Status 404 returned error can't find the container with id bf2f5277657e1cfd9b58f6407875bf2dcf8ba455b0f127510ce11493bcc9154e Oct 30 06:13:32.027297 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:32.027258 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" event={"ID":"2753399b-3bdc-4c2a-b41b-c941c1af7271","Type":"ContainerStarted","Data":"bf2f5277657e1cfd9b58f6407875bf2dcf8ba455b0f127510ce11493bcc9154e"} Oct 30 06:13:34.045844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:34.045792 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" event={"ID":"2753399b-3bdc-4c2a-b41b-c941c1af7271","Type":"ContainerStarted","Data":"40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece"} Oct 30 06:13:34.046315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:34.045884 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:34.098127 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:34.098077 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" podStartSLOduration=1.328028405 podStartE2EDuration="3.09806366s" totalImagesPullingTime="1.770035255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:34.095720074 +0000 UTC m=+2520.671407131" watchObservedRunningTime="2026-10-30 06:13:34.09806366 +0000 UTC m=+2520.673750714" Oct 30 06:13:37.623984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:37.623724 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:13:38.388241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:38.388206 5565 scope.go:122] "RemoveContainer" containerID="48c41e483e344685343648cd456300d987a8bf733924a5ae702ad7299e80fe17" Oct 30 06:13:38.399141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:38.399115 5565 scope.go:122] "RemoveContainer" containerID="afa8c26a29639effff852ca3b26563ec7c6b71ac0bd8c3ebb9420e711cd632eb" Oct 30 06:13:45.057709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:45.057675 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:13:47.534509 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:13:47.534469 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:13:47.861113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:47.861032 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg"] Oct 30 06:13:47.868382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:47.868352 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:47.894417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:47.894389 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg"] Oct 30 06:13:47.902420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:47.902395 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-370dfa-openshift-default-dockercfg-pj6m5\"" Oct 30 06:13:48.018901 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.018863 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.019232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.019207 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.019371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.019355 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.043552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.039609 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.043552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.039723 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.043552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.039778 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cb8b\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.043552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.039889 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.043552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.040005 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.043552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.040040 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.142194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.142100 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.142432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.142407 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.142614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.142586 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.142814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.142796 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.142987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.142967 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cb8b\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.143167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.143152 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.144704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.144683 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.144954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.144216 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.144954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.143599 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.145154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.144873 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.145339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.145314 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.145599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.145579 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.146262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.145887 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.146417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.145946 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.147862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.147836 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.150024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.149990 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.171114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.171086 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.172724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.172698 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cb8b\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b\") pod \"e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.193282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.193244 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:48.578114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.578085 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg"] Oct 30 06:13:48.578767 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:13:48.578739 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd078081e_660d_4c3a_8c14_dd61d34f9e82.slice/crio-95ac1663df6d7dd4ac185b68d113a2ed9cd2adc81a70af47c7a3c660b9fe23d0 WatchSource:0}: Error finding container 95ac1663df6d7dd4ac185b68d113a2ed9cd2adc81a70af47c7a3c660b9fe23d0: Status 404 returned error can't find the container with id 95ac1663df6d7dd4ac185b68d113a2ed9cd2adc81a70af47c7a3c660b9fe23d0 Oct 30 06:13:48.581574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.581504 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:13:48.581693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.581625 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:13:48.581693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:48.581672 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:13:49.248547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:49.247582 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" event={"ID":"d078081e-660d-4c3a-8c14-dd61d34f9e82","Type":"ContainerStarted","Data":"46a0d14d45a7a0a93bbc4ef99a99f38b0be144615d59506eed0d9d2b38504ede"} Oct 30 06:13:49.248547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:49.247649 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" event={"ID":"d078081e-660d-4c3a-8c14-dd61d34f9e82","Type":"ContainerStarted","Data":"95ac1663df6d7dd4ac185b68d113a2ed9cd2adc81a70af47c7a3c660b9fe23d0"} Oct 30 06:13:49.337275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:49.337226 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podStartSLOduration=2.337210502 podStartE2EDuration="2.337210502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:49.334243843 +0000 UTC m=+2535.909930898" watchObservedRunningTime="2026-10-30 06:13:49.337210502 +0000 UTC m=+2535.912897624" Oct 30 06:13:50.194582 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:50.194546 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:50.200700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:50.200675 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:50.256245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:50.256219 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:50.257102 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:50.257082 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:13:51.147409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:51.147375 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp"] Oct 30 06:13:53.679424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.679386 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst"] Oct 30 06:13:53.685425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.685405 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:53.701382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.701356 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst"] Oct 30 06:13:53.768550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.761810 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-370dfa\"" Oct 30 06:13:53.781608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.770258 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-370dfa-dockercfg-7dw2b\"" Oct 30 06:13:53.874318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.859475 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cbkx\" (UniqueName: \"kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:53.874318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.859576 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:53.965629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.964835 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cbkx\" (UniqueName: \"kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:53.965629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.964908 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:53.966751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.966728 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:53.990518 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:53.990481 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cbkx\" (UniqueName: \"kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:54.002288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.002254 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:54.193749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.193715 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4"] Oct 30 06:13:54.199517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.199497 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.216503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.216451 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-2f1e4c-openshift-default-dockercfg-r26nk\"" Oct 30 06:13:54.226013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.225988 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4"] Oct 30 06:13:54.332585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.332555 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst"] Oct 30 06:13:54.373010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.372980 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373016 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373043 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373199 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373229 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373246 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k77hd\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373266 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373346 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.373489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.373475 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474323 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474373 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474390 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k77hd\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474506 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474586 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474784 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474801 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474905 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474913 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.474982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474947 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.475113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.474989 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.475113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.475053 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.475268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.475242 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.475677 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.475623 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.476706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.476684 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.477169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.477151 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.501324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.501298 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.501512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.501492 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k77hd\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd\") pod \"e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.516267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.516245 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:54.855799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.855773 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4"] Oct 30 06:13:54.856285 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:13:54.856260 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefb6f096_bce0_47bc_bd26_6049d733d934.slice/crio-d7b9cb916408777fd8b3e6671d18670879a2173f568907c726d4714bb2e12c45 WatchSource:0}: Error finding container d7b9cb916408777fd8b3e6671d18670879a2173f568907c726d4714bb2e12c45: Status 404 returned error can't find the container with id d7b9cb916408777fd8b3e6671d18670879a2173f568907c726d4714bb2e12c45 Oct 30 06:13:54.859126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.859089 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:13:54.859250 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.859172 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:13:54.859250 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:54.859220 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:13:55.300319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.300281 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" event={"ID":"efb6f096-bce0-47bc-bd26-6049d733d934","Type":"ContainerStarted","Data":"0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507"} Oct 30 06:13:55.300572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.300352 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" event={"ID":"efb6f096-bce0-47bc-bd26-6049d733d934","Type":"ContainerStarted","Data":"d7b9cb916408777fd8b3e6671d18670879a2173f568907c726d4714bb2e12c45"} Oct 30 06:13:55.302164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.302138 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" event={"ID":"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce","Type":"ContainerStarted","Data":"945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da"} Oct 30 06:13:55.302308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.302180 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" event={"ID":"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce","Type":"ContainerStarted","Data":"2698bedf7a43e65442248fcf37b64d5302db4b0c25a348caf6ccd5a64d531c5e"} Oct 30 06:13:55.302308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.302207 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:13:55.351693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.351646 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podStartSLOduration=1.351632152 podStartE2EDuration="1.351632152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:55.351199115 +0000 UTC m=+2541.926886170" watchObservedRunningTime="2026-10-30 06:13:55.351632152 +0000 UTC m=+2541.927319206" Oct 30 06:13:55.398574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.398455 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" podStartSLOduration=2.39843734 podStartE2EDuration="2.39843734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:13:55.397589447 +0000 UTC m=+2541.973276502" watchObservedRunningTime="2026-10-30 06:13:55.39843734 +0000 UTC m=+2541.974124396" Oct 30 06:13:55.516595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.516561 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:13:55.518282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.518254 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:13:55.518435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:55.518308 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:13:56.148559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.148490 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" podUID="313637b4-a045-4e66-9064-7c59b12a1da2" containerName="payload-pre-processing" containerID="cri-o://0e25ab6f1f4fc005b7f393f047fae0c120a200fabcf29b37da1384451bce4bc5" gracePeriod=25 Oct 30 06:13:56.313073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.313039 5565 generic.go:356] "Generic (PLEG): container finished" podID="313637b4-a045-4e66-9064-7c59b12a1da2" containerID="0e25ab6f1f4fc005b7f393f047fae0c120a200fabcf29b37da1384451bce4bc5" exitCode=0 Oct 30 06:13:56.313338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.313079 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" event={"ID":"313637b4-a045-4e66-9064-7c59b12a1da2","Type":"ContainerDied","Data":"0e25ab6f1f4fc005b7f393f047fae0c120a200fabcf29b37da1384451bce4bc5"} Oct 30 06:13:56.516740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.516712 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:13:56.516854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.516764 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:13:56.520739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.520717 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:13:56.710769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.710732 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk\" (UniqueName: \"kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk\") pod \"313637b4-a045-4e66-9064-7c59b12a1da2\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " Oct 30 06:13:56.710992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.710973 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume\") pod \"313637b4-a045-4e66-9064-7c59b12a1da2\" (UID: \"313637b4-a045-4e66-9064-7c59b12a1da2\") " Oct 30 06:13:56.711393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.711363 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume" pod "313637b4-a045-4e66-9064-7c59b12a1da2" (UID: "313637b4-a045-4e66-9064-7c59b12a1da2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:13:56.711713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.711690 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/313637b4-a045-4e66-9064-7c59b12a1da2-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:13:56.713171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.713142 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk" pod "313637b4-a045-4e66-9064-7c59b12a1da2" (UID: "313637b4-a045-4e66-9064-7c59b12a1da2"). InnerVolumeSpecName "kube-api-access-c4qmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:13:56.813134 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:56.813095 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4qmk\" (UniqueName: \"kubernetes.io/projected/313637b4-a045-4e66-9064-7c59b12a1da2-kube-api-access-c4qmk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:13:57.323052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.323026 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" Oct 30 06:13:57.323638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.323399 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp" event={"ID":"313637b4-a045-4e66-9064-7c59b12a1da2","Type":"ContainerDied","Data":"0a14a533c29eeb5190c2d17b536bae0dcfc3ca4bb0813ab69c47489c90759ee2"} Oct 30 06:13:57.323638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.323447 5565 scope.go:122] "RemoveContainer" containerID="0e25ab6f1f4fc005b7f393f047fae0c120a200fabcf29b37da1384451bce4bc5" Oct 30 06:13:57.386513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.386481 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp"] Oct 30 06:13:57.399091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.399063 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-54dcfc75f9-vsnlp"] Oct 30 06:13:57.517490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.517452 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:13:57.517712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.517507 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:13:57.586438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:57.586358 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="313637b4-a045-4e66-9064-7c59b12a1da2" path="/var/lib/kubelet/pods/313637b4-a045-4e66-9064-7c59b12a1da2/volumes" Oct 30 06:13:58.516602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:58.516567 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:13:58.517078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:58.516621 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:13:59.516568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:59.516505 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:13:59.516757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:13:59.516587 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:14:00.516986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:00.516953 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:14:00.517373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:00.517003 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:14:01.516620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:01.516580 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" start-of-body= Oct 30 06:14:01.516824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:01.516653 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.239:15021/healthz/ready\": dial tcp 10.134.0.239:15021: connect: connection refused" Oct 30 06:14:01.945054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:01.944973 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw"] Oct 30 06:14:02.661909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:02.661873 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:14:02.662827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:02.662803 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:14:02.664067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:02.664041 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:14:06.316235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:06.316207 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:14:06.945488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:06.945452 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" podUID="08863796-4736-4be3-8d67-6bf865f69983" containerName="payload-pre-processing" containerID="cri-o://75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1" gracePeriod=25 Oct 30 06:14:07.226586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.226559 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:14:07.353953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.353921 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6\" (UniqueName: \"kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6\") pod \"08863796-4736-4be3-8d67-6bf865f69983\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " Oct 30 06:14:07.354424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.354090 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume\") pod \"08863796-4736-4be3-8d67-6bf865f69983\" (UID: \"08863796-4736-4be3-8d67-6bf865f69983\") " Oct 30 06:14:07.354503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.354476 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume" pod "08863796-4736-4be3-8d67-6bf865f69983" (UID: "08863796-4736-4be3-8d67-6bf865f69983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:07.356008 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.355986 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6" pod "08863796-4736-4be3-8d67-6bf865f69983" (UID: "08863796-4736-4be3-8d67-6bf865f69983"). InnerVolumeSpecName "kube-api-access-r5hg6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:07.408042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.408011 5565 generic.go:356] "Generic (PLEG): container finished" podID="08863796-4736-4be3-8d67-6bf865f69983" containerID="75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1" exitCode=0 Oct 30 06:14:07.408178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.408058 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" Oct 30 06:14:07.408178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.408097 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" event={"ID":"08863796-4736-4be3-8d67-6bf865f69983","Type":"ContainerDied","Data":"75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1"} Oct 30 06:14:07.408285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.408177 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw" event={"ID":"08863796-4736-4be3-8d67-6bf865f69983","Type":"ContainerDied","Data":"28a1240647f16dd87182ccf890e4156ffb1e3746257538ef4559ced92e65952a"} Oct 30 06:14:07.408285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.408207 5565 scope.go:122] "RemoveContainer" containerID="75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1" Oct 30 06:14:07.419931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.419914 5565 scope.go:122] "RemoveContainer" containerID="75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1" Oct 30 06:14:07.420185 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:14:07.420163 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1\": container with ID starting with 75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1 not found: ID does not exist" containerID="75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1" Oct 30 06:14:07.420231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.420196 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1"} err="failed to get container status \"75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1\": rpc error: code = NotFound desc = could not find container \"75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1\": container with ID starting with 75d32dc50f1cfa6d3497e723a8acb02efd347cb60e798d76afeee01d069ad3f1 not found: ID does not exist" Oct 30 06:14:07.456156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.456089 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5hg6\" (UniqueName: \"kubernetes.io/projected/08863796-4736-4be3-8d67-6bf865f69983-kube-api-access-r5hg6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:07.456156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.456127 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08863796-4736-4be3-8d67-6bf865f69983-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:07.470221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.470199 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw"] Oct 30 06:14:07.474320 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.474299 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-7684f667f5-7hrmw"] Oct 30 06:14:07.570396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:07.570364 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08863796-4736-4be3-8d67-6bf865f69983" path="/var/lib/kubelet/pods/08863796-4736-4be3-8d67-6bf865f69983/volumes" Oct 30 06:14:17.533120 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:14:17.533083 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:14:22.142331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:22.142297 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7"] Oct 30 06:14:22.142749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:22.142502 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" containerID="cri-o://21f242929e742ff787fc712c63906d0039095df46abbe57f997df5598ca6064a" gracePeriod=30 Oct 30 06:14:25.029316 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:25.029280 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:14:27.700763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:27.697053 5565 generic.go:356] "Generic (PLEG): container finished" podID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerID="21f242929e742ff787fc712c63906d0039095df46abbe57f997df5598ca6064a" exitCode=0 Oct 30 06:14:27.735271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:27.735229 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" event={"ID":"b26ccd6a-916a-4f95-aa97-91dfe6e3c889","Type":"ContainerDied","Data":"21f242929e742ff787fc712c63906d0039095df46abbe57f997df5598ca6064a"} Oct 30 06:14:27.735478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:27.735315 5565 scope.go:122] "RemoveContainer" containerID="480564cdd24754f2d1cc269989c26c006458b996998983e0fc0a368254a5039c" Oct 30 06:14:28.008729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.008695 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 06:14:28.097606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.097029 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.097606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.097151 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.097606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.097229 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.097606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.097288 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.104548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.098302 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.104548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.098442 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.104548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.098591 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.104548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.098758 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.104548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.099053 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs\") pod \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\" (UID: \"b26ccd6a-916a-4f95-aa97-91dfe6e3c889\") " Oct 30 06:14:28.104548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.101413 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:28.106067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.105877 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.113613 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.113570 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:28.114657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.114069 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:14:28.114657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.114609 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:28.121269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.121097 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:28.121269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.121214 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:28.121624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.121592 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:14:28.124701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.124674 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "kube-api-access-vqsnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:14:28.126957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.126920 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo" pod "b26ccd6a-916a-4f95-aa97-91dfe6e3c889" (UID: "b26ccd6a-916a-4f95-aa97-91dfe6e3c889"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:14:28.207344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207313 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207342 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqsnr\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-kube-api-access-vqsnr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207351 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207361 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207370 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207381 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207390 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.207601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.207399 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b26ccd6a-916a-4f95-aa97-91dfe6e3c889-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:14:28.707581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.707515 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" Oct 30 06:14:28.708023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.707507 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7" event={"ID":"b26ccd6a-916a-4f95-aa97-91dfe6e3c889","Type":"ContainerDied","Data":"b3ca18681b34d03e5b4aa0710b0e00698dcec837d18987106d158e3eb3de83d1"} Oct 30 06:14:28.708023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.707687 5565 scope.go:122] "RemoveContainer" containerID="21f242929e742ff787fc712c63906d0039095df46abbe57f997df5598ca6064a" Oct 30 06:14:28.781617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.781581 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7"] Oct 30 06:14:28.795398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:28.795362 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-zsdz7"] Oct 30 06:14:29.572199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:29.572154 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" path="/var/lib/kubelet/pods/b26ccd6a-916a-4f95-aa97-91dfe6e3c889/volumes" Oct 30 06:14:33.718657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.718419 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl"] Oct 30 06:14:33.720543 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.720297 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" Oct 30 06:14:33.720716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.720698 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" Oct 30 06:14:33.720911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.720899 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" Oct 30 06:14:33.721006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.720995 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" Oct 30 06:14:33.721088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721078 5565 cpu_manager.go:414] "containerMap: removing container" podUID="313637b4-a045-4e66-9064-7c59b12a1da2" containerName="payload-pre-processing" Oct 30 06:14:33.721163 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721153 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="313637b4-a045-4e66-9064-7c59b12a1da2" containerName="payload-pre-processing" Oct 30 06:14:33.721243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721234 5565 cpu_manager.go:414] "containerMap: removing container" podUID="08863796-4736-4be3-8d67-6bf865f69983" containerName="payload-pre-processing" Oct 30 06:14:33.721317 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721308 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08863796-4736-4be3-8d67-6bf865f69983" containerName="payload-pre-processing" Oct 30 06:14:33.721698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721679 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" Oct 30 06:14:33.721840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721826 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="313637b4-a045-4e66-9064-7c59b12a1da2" containerName="payload-pre-processing" Oct 30 06:14:33.721932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.721921 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="08863796-4736-4be3-8d67-6bf865f69983" containerName="payload-pre-processing" Oct 30 06:14:33.726431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.726415 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl"] Oct 30 06:14:33.726608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.726556 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:33.744169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.744132 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-36ae27\"" Oct 30 06:14:33.757862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.757831 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-36ae27-dockercfg-v45qz\"" Oct 30 06:14:33.764540 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.764495 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd"] Oct 30 06:14:33.765448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.765427 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b26ccd6a-916a-4f95-aa97-91dfe6e3c889" containerName="istio-proxy" Oct 30 06:14:33.769153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.769134 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:33.784738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.784708 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd"] Oct 30 06:14:33.833295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.833257 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:33.833631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.833363 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5njqs\" (UniqueName: \"kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:33.935148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.935111 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhkb\" (UniqueName: \"kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb\") pod \"payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:33.935307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.935158 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:33.935307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.935275 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5njqs\" (UniqueName: \"kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:33.935386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.935362 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:33.935993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.935975 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:33.960846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:33.960820 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5njqs\" (UniqueName: \"kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs\") pod \"payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:34.037020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.036990 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhkb\" (UniqueName: \"kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb\") pod \"payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:34.037151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.037104 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:34.037809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.037789 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:34.054362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.054339 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:34.062056 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.062028 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhkb\" (UniqueName: \"kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb\") pod \"payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:34.090398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.090370 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:34.364851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.364821 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl"] Oct 30 06:14:34.365122 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:14:34.365098 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod403ff440_ad88_4ce8_95b4_1aa3f07cb389.slice/crio-ed9460421aa0251168132852dbb8883d0517afb5bf5c5c37a114437c4360e226 WatchSource:0}: Error finding container ed9460421aa0251168132852dbb8883d0517afb5bf5c5c37a114437c4360e226: Status 404 returned error can't find the container with id ed9460421aa0251168132852dbb8883d0517afb5bf5c5c37a114437c4360e226 Oct 30 06:14:34.459275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.448198 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd"] Oct 30 06:14:34.469560 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:14:34.466091 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a2fd88_6c33_4ebf_afba_0f8a87720e92.slice/crio-1dcc539138dfe996953023a8396496e28acedf6123afae3053ab67d2d0e407be WatchSource:0}: Error finding container 1dcc539138dfe996953023a8396496e28acedf6123afae3053ab67d2d0e407be: Status 404 returned error can't find the container with id 1dcc539138dfe996953023a8396496e28acedf6123afae3053ab67d2d0e407be Oct 30 06:14:34.763008 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.762967 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" event={"ID":"b4a2fd88-6c33-4ebf-afba-0f8a87720e92","Type":"ContainerStarted","Data":"306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347"} Oct 30 06:14:34.763610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.763405 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" event={"ID":"b4a2fd88-6c33-4ebf-afba-0f8a87720e92","Type":"ContainerStarted","Data":"1dcc539138dfe996953023a8396496e28acedf6123afae3053ab67d2d0e407be"} Oct 30 06:14:34.763610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.763438 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:34.765433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.765409 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" event={"ID":"403ff440-ad88-4ce8-95b4-1aa3f07cb389","Type":"ContainerStarted","Data":"dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784"} Oct 30 06:14:34.765623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.765451 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" event={"ID":"403ff440-ad88-4ce8-95b4-1aa3f07cb389","Type":"ContainerStarted","Data":"ed9460421aa0251168132852dbb8883d0517afb5bf5c5c37a114437c4360e226"} Oct 30 06:14:34.765623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.765478 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:34.812217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:34.812170 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" podStartSLOduration=1.8121578760000001 podStartE2EDuration="1.812157876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:34.808385776 +0000 UTC m=+2581.384072830" watchObservedRunningTime="2026-10-30 06:14:34.812157876 +0000 UTC m=+2581.387844930" Oct 30 06:14:38.436794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:38.436759 5565 scope.go:122] "RemoveContainer" containerID="20aa1093cc7b44402782b3ea93dc633f63139d8844710cf7f5f633ee465ad166" Oct 30 06:14:38.447281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:38.447260 5565 scope.go:122] "RemoveContainer" containerID="4dfbe389b2816d47e37cc4e45a96e07636b86f8bb37c3f1ed516518c7e455d7c" Oct 30 06:14:43.067698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.063677 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" podStartSLOduration=10.063653137 podStartE2EDuration="10.063653137s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:34.86704264 +0000 UTC m=+2581.442729702" watchObservedRunningTime="2026-10-30 06:14:43.063653137 +0000 UTC m=+2589.639340191" Oct 30 06:14:43.067698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.066616 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb"] Oct 30 06:14:43.085553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.078509 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.093548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.089688 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb"] Oct 30 06:14:43.097551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.096216 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-ebe61e\"" Oct 30 06:14:43.097551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.096345 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-ebe61e-dockercfg-64v9n\"" Oct 30 06:14:43.162551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.149909 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj5bt\" (UniqueName: \"kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.162551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.150603 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.237449 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.237411 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm"] Oct 30 06:14:43.243163 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.243140 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.272568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.265578 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj5bt\" (UniqueName: \"kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.272568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.265736 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.272568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.266863 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.284370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.284341 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm"] Oct 30 06:14:43.294280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.294244 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj5bt\" (UniqueName: \"kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt\") pod \"payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.367108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.367042 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.367242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.367110 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66\") pod \"payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.450144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.450115 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:43.469130 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.469097 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.469229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.469182 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66\") pod \"payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.469990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.469967 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.495118 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.495087 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66\") pod \"payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.581289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.581262 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:43.761429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.761406 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb"] Oct 30 06:14:43.761579 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:14:43.761554 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d16eafb_cedd_4a48_9e6e_77b7bf42efd8.slice/crio-6c6348c3d44fac676d4e74216044676b646e23a461775874df5d775c0d57a81a WatchSource:0}: Error finding container 6c6348c3d44fac676d4e74216044676b646e23a461775874df5d775c0d57a81a: Status 404 returned error can't find the container with id 6c6348c3d44fac676d4e74216044676b646e23a461775874df5d775c0d57a81a Oct 30 06:14:43.855639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.855605 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" event={"ID":"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8","Type":"ContainerStarted","Data":"6c6348c3d44fac676d4e74216044676b646e23a461775874df5d775c0d57a81a"} Oct 30 06:14:43.907648 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:14:43.907615 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885d6370_e620_4728_af2d_37dc726d58bc.slice/crio-71d329704b929c2c5f899ee62604e0da05313192d7caa6e105b315d5dcc5428c WatchSource:0}: Error finding container 71d329704b929c2c5f899ee62604e0da05313192d7caa6e105b315d5dcc5428c: Status 404 returned error can't find the container with id 71d329704b929c2c5f899ee62604e0da05313192d7caa6e105b315d5dcc5428c Oct 30 06:14:43.907795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:43.907668 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm"] Oct 30 06:14:44.866263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.866223 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" event={"ID":"885d6370-e620-4728-af2d-37dc726d58bc","Type":"ContainerStarted","Data":"a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275"} Oct 30 06:14:44.866836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.866289 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" event={"ID":"885d6370-e620-4728-af2d-37dc726d58bc","Type":"ContainerStarted","Data":"71d329704b929c2c5f899ee62604e0da05313192d7caa6e105b315d5dcc5428c"} Oct 30 06:14:44.866836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.866333 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:44.868169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.868137 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" event={"ID":"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8","Type":"ContainerStarted","Data":"f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528"} Oct 30 06:14:44.868316 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.868222 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:44.913489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.913443 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" podStartSLOduration=1.9134303419999998 podStartE2EDuration="1.913430342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:44.911054698 +0000 UTC m=+2591.486741753" watchObservedRunningTime="2026-10-30 06:14:44.913430342 +0000 UTC m=+2591.489117395" Oct 30 06:14:44.959292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:44.959234 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" podStartSLOduration=1.9592175840000001 podStartE2EDuration="1.959217584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:44.955583367 +0000 UTC m=+2591.531270421" watchObservedRunningTime="2026-10-30 06:14:44.959217584 +0000 UTC m=+2591.534904643" Oct 30 06:14:45.778052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:45.778023 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:14:45.778266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:45.778139 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:14:47.537998 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:14:47.537960 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:14:47.764783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:47.764751 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7"] Oct 30 06:14:47.799320 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:47.799246 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw"] Oct 30 06:14:52.765708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:52.765652 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerName="payload-processing" containerID="cri-o://566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25" gracePeriod=25 Oct 30 06:14:52.800215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:52.800180 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" podUID="81e69693-69c0-4d07-867e-8d385f41c342" containerName="payload-pre-processing" containerID="cri-o://434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791" gracePeriod=25 Oct 30 06:14:54.183270 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:54.183232 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:14:54.183723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:54.183232 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" podUID="81e69693-69c0-4d07-867e-8d385f41c342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:14:55.883038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:55.883008 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:14:55.883427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:55.883074 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:14:57.398028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.397993 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68"] Oct 30 06:14:57.402713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.402691 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.441751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.441718 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-95498c-openshift-default-dockercfg-hvq8v\"" Oct 30 06:14:57.446774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.446752 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68"] Oct 30 06:14:57.480470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.480434 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97cp9\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.480653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.480636 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.480736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.480692 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.480805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.480737 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.480942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.480919 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.481052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.480992 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.481170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.481140 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.481235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.481195 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.481327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.481300 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.582904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.582875 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.582909 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.582936 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.582982 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97cp9\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583095 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583265 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583324 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583389 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583421 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583489 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583719 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583820 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583778 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583820 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.583946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.583923 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.585418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.585395 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.585584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.585564 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.607704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.607679 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.608011 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.607991 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97cp9\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9\") pod \"e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:57.718085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:57.718023 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:58.120846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:58.120810 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68"] Oct 30 06:14:58.144179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:58.143079 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:14:58.144179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:58.143188 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:14:58.144179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:58.143229 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:14:58.382939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:58.382260 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d"] Oct 30 06:14:58.382939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:58.382514 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" containerID="cri-o://21ffeb60497120eb2906eff3a2fa46b4694f90cce39f1e19c6761d25eda0dfa1" gracePeriod=30 Oct 30 06:14:59.083425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:59.083386 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" event={"ID":"d6437fe8-d09c-419d-ac5b-33f689da1554","Type":"ContainerStarted","Data":"a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4"} Oct 30 06:14:59.083855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:59.083446 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" event={"ID":"d6437fe8-d09c-419d-ac5b-33f689da1554","Type":"ContainerStarted","Data":"89f797408788950eac7910f2d31d8a9f768b5c0dced080366386b493668333eb"} Oct 30 06:14:59.137114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:59.137060 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podStartSLOduration=2.137046353 podStartE2EDuration="2.137046353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:14:59.133182468 +0000 UTC m=+2605.708869525" watchObservedRunningTime="2026-10-30 06:14:59.137046353 +0000 UTC m=+2605.712733451" Oct 30 06:14:59.725556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:59.723457 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:14:59.725556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:59.723644 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 06:14:59.725556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:14:59.723698 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 06:15:00.180549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.166578 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb"] Oct 30 06:15:00.186980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.186938 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.197611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.197580 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p"] Oct 30 06:15:00.212788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.212760 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:15:00.217021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.216998 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg"] Oct 30 06:15:00.217448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.217356 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.227005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.226982 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb"] Oct 30 06:15:00.227289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.227272 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f"] Oct 30 06:15:00.228655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.227218 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.246665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.246640 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl"] Oct 30 06:15:00.247452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.247333 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.305551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.304513 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p"] Oct 30 06:15:00.305551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.304587 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg"] Oct 30 06:15:00.305551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.304618 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6"] Oct 30 06:15:00.306862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.306840 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328212 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mp9l\" (UniqueName: \"kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328294 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328377 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328540 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2f5\" (UniqueName: \"kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328608 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328633 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd6zc\" (UniqueName: \"kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328660 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6hsf\" (UniqueName: \"kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.328997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.328806 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.334402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.334256 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg"] Oct 30 06:15:00.334755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.334350 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.403743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.403714 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.422936 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.402887 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf"] Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435192 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzhd2\" (UniqueName: \"kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435252 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mp9l\" (UniqueName: \"kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435325 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435400 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435494 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5cwb\" (UniqueName: \"kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435622 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2f5\" (UniqueName: \"kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435668 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435719 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435746 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd6zc\" (UniqueName: \"kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.435775 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6hsf\" (UniqueName: \"kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.436062 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.436142 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.436584 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.436873 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.437499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.437226 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.443282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.443256 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.516484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.516448 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f"] Oct 30 06:15:00.516765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.516749 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl"] Oct 30 06:15:00.516922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.516911 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r"] Oct 30 06:15:00.523573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.516801 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:00.524773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.524742 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6hsf\" (UniqueName: \"kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.524905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.524874 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2f5\" (UniqueName: \"kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5\") pod \"maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.525551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.525511 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd6zc\" (UniqueName: \"kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.525728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.525707 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6"] Oct 30 06:15:00.525789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.525753 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b"] Oct 30 06:15:00.526123 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.526106 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:00.530115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.530089 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mp9l\" (UniqueName: \"kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.532138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.532117 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf"] Oct 30 06:15:00.532229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.532178 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv"] Oct 30 06:15:00.532392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.532377 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:00.538359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.538335 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzhd2\" (UniqueName: \"kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.539425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.538545 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.539425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.538595 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5cwb\" (UniqueName: \"kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.539425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.538702 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgz7z\" (UniqueName: \"kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.539425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.538743 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.539425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.538856 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.539425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.539394 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.539802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.539777 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.545365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.545345 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:00.546314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.546295 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk"] Oct 30 06:15:00.546437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.546403 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:00.552522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.552504 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r"] Oct 30 06:15:00.552654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.552564 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk"] Oct 30 06:15:00.552654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.552648 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:00.667586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.634292 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:00.667586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.634751 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:00.667586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.640307 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.667586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.640451 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgz7z\" (UniqueName: \"kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.667586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.646756 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.727750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.719252 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:00.727750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.727652 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 06:15:00.727750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.727705 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 06:15:00.919082 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.919043 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgz7z\" (UniqueName: \"kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z\") pod \"maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:00.920410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.920263 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzhd2\" (UniqueName: \"kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:00.920662 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.920505 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5cwb\" (UniqueName: \"kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:00.926234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.926202 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg"] Oct 30 06:15:00.933335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.933238 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tfjl\" (UniqueName: \"kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl\") pod \"maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:00.933684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.933556 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:00.933990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.933825 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:00.934197 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.934180 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:00.934776 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.934592 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65dn8\" (UniqueName: \"kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:00.934938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.934919 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:00.936578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.936549 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5v2x\" (UniqueName: \"kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x\") pod \"maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:00.936741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.936723 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:00.936914 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.936896 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hkkm\" (UniqueName: \"kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:00.936983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.936968 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jk9z\" (UniqueName: \"kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z\") pod \"maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:00.944369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:00.944335 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv"] Oct 30 06:15:01.290952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.290920 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:01.328901 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.306795 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.307263 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.307814 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.329989 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hkkm\" (UniqueName: \"kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.308703 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.347201 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jk9z\" (UniqueName: \"kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z\") pod \"maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.347443 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tfjl\" (UniqueName: \"kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl\") pod \"maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.347633 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.347666 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.349223 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.349335 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65dn8\" (UniqueName: \"kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.349373 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.349423 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5v2x\" (UniqueName: \"kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x\") pod \"maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.351115 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.351729 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.352565 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:01.400584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.353101 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:01.519788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.519364 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5v2x\" (UniqueName: \"kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x\") pod \"maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:01.553217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.539395 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jk9z\" (UniqueName: \"kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z\") pod \"maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:01.553217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.543587 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hkkm\" (UniqueName: \"kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm\") pod \"maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:01.553217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.544275 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65dn8\" (UniqueName: \"kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8\") pod \"maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:01.553217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.544909 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b"] Oct 30 06:15:01.553217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.545704 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tfjl\" (UniqueName: \"kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl\") pod \"maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:01.610925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.610891 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:01.624382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.624356 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:01.626286 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.624915 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:01.719766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.719214 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:01.722236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.722188 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 06:15:01.722375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.722250 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 06:15:01.741068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.741043 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb"] Oct 30 06:15:01.824577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:01.824509 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:02.198030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.172542 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p"] Oct 30 06:15:02.444823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.443907 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg"] Oct 30 06:15:02.534768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.534727 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f"] Oct 30 06:15:02.719997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.719904 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 06:15:02.719997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.719952 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 06:15:02.731085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.731055 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerStarted","Data":"4176e54a961ea51576d974809fbf2b96626c30f1efa51f68e9eb6f4f49fd9d22"} Oct 30 06:15:02.738133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.738103 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerStarted","Data":"fcdcaddc3098c20d025f463b1e016496412361a7745692afcd5d7f413c70cd1f"} Oct 30 06:15:02.740251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.740173 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerStarted","Data":"d5117ee847d0a89de2b37966cb9cfc0de61bc683b8028ed1bb789a334c413743"} Oct 30 06:15:02.747780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.747750 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" event={"ID":"633df5e9-b8cf-480a-9b49-1a013f55e729","Type":"ContainerStarted","Data":"23a34aa5abf610a919cbc0105a36f160fcd0c4b8d9b093f851a29ea8189a7022"} Oct 30 06:15:02.794959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.794919 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r"] Oct 30 06:15:02.795247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.795228 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6"] Oct 30 06:15:02.915153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:02.913211 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg"] Oct 30 06:15:03.119822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.119718 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl"] Oct 30 06:15:03.215812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.214699 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf"] Oct 30 06:15:03.227801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.227727 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b"] Oct 30 06:15:03.319183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.319130 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk"] Oct 30 06:15:03.323660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.323623 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv"] Oct 30 06:15:03.719553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.718509 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 06:15:03.719553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.718583 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 06:15:03.845545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.845490 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerStarted","Data":"27b3ba914ef6b4f9c251e7b6964ccf8854d15b89546ce7e965728a4cde5c5b2b"} Oct 30 06:15:03.848644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.848612 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerStarted","Data":"3b72b0383882f7e16bc7bda56f8a70b2dd8dfdc6659fb2fdffb42320975b77dd"} Oct 30 06:15:03.928074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.928029 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" event={"ID":"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b","Type":"ContainerStarted","Data":"22863b802198e812922cbd4b5d66d93635964c94043e26686aa7fbf7a5c03551"} Oct 30 06:15:03.929681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.928725 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerStarted","Data":"5f2e26ca60358d42c180479aefe11cbc07e3dd5510a80652834df12e36ba1da2"} Oct 30 06:15:03.929681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.929177 5565 scope.go:122] "RemoveContainer" containerID="5f2e26ca60358d42c180479aefe11cbc07e3dd5510a80652834df12e36ba1da2" Oct 30 06:15:03.944412 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.943463 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" podStartSLOduration=3.943445496 podStartE2EDuration="3.943445496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:03.940002574 +0000 UTC m=+2610.515689628" watchObservedRunningTime="2026-10-30 06:15:03.943445496 +0000 UTC m=+2610.519132580" Oct 30 06:15:03.947564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.947520 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerStarted","Data":"e98e6ec2ed41c1dedd1041653e9b931c24336349b47784473b9af9d4ea000200"} Oct 30 06:15:03.948482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.948453 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerStarted","Data":"58a6565927d0fa1a89f0960b19b64c4145aac239f3904be86189f03749ac8722"} Oct 30 06:15:03.950418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.950389 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerStarted","Data":"4a9a37c62e214595b0bddb261f5c83600abefb34d68373a2d5a59c674d1b3090"} Oct 30 06:15:03.978963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.977743 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerStarted","Data":"58519c6528312174eb3d57ba24a1c2671819be1bdfb3d506ca7a1f839dc00441"} Oct 30 06:15:03.978963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:03.977805 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerStarted","Data":"84c42e62b53a3f8a44aa820abdfd97546efc19bb4610d07b81a5d8ee5613eb07"} Oct 30 06:15:04.025907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.025873 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerStarted","Data":"8435c949bdded446d03e89cc3e02e9ed03455015424f2656cb0aec24fc6df6b4"} Oct 30 06:15:04.026053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.025938 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerStarted","Data":"1421a980d9d80b1967adb2248ae7ffed9251b19e4709c1730afe1991c77d18e7"} Oct 30 06:15:04.029687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.029496 5565 generic.go:356] "Generic (PLEG): container finished" podID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerID="599f6f23ec01fa925c7ea9ef09c41a08dd23872361b08399a005c471c4a540b0" exitCode=60 Oct 30 06:15:04.029687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.029631 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" event={"ID":"633df5e9-b8cf-480a-9b49-1a013f55e729","Type":"ContainerDied","Data":"599f6f23ec01fa925c7ea9ef09c41a08dd23872361b08399a005c471c4a540b0"} Oct 30 06:15:04.030351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.029940 5565 scope.go:122] "RemoveContainer" containerID="599f6f23ec01fa925c7ea9ef09c41a08dd23872361b08399a005c471c4a540b0" Oct 30 06:15:04.031404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.031371 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerStarted","Data":"05d0a0d00241a8d850d7717db810da885bf345422e6e7c09740abc03bcaa8b38"} Oct 30 06:15:04.033468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.033446 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" event={"ID":"196a2b1b-619c-416b-a839-e3b1fd15699f","Type":"ContainerStarted","Data":"98c5b404e0ed356cedec231f28cb3a99cc748afe421699b64a103d4eb05516ee"} Oct 30 06:15:04.036374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.036346 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerStarted","Data":"77379c0c17f3fffc8decbb6ec7777a3df722a59bfdadfb7f4368144d16a60ac2"} Oct 30 06:15:04.036585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.036569 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerStarted","Data":"cfbf88529fe10b9c14133030c8ae5d8bfc43d32efef99025a3726123dab0eda7"} Oct 30 06:15:04.038982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.038958 5565 generic.go:356] "Generic (PLEG): container finished" podID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerID="21ffeb60497120eb2906eff3a2fa46b4694f90cce39f1e19c6761d25eda0dfa1" exitCode=0 Oct 30 06:15:04.039072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.039029 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" event={"ID":"67c3d184-a561-4c40-a950-6e93e7ae48de","Type":"ContainerDied","Data":"21ffeb60497120eb2906eff3a2fa46b4694f90cce39f1e19c6761d25eda0dfa1"} Oct 30 06:15:04.088955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.067807 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" podStartSLOduration=4.067786356 podStartE2EDuration="4.067786356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.049340388 +0000 UTC m=+2610.625027443" watchObservedRunningTime="2026-10-30 06:15:04.067786356 +0000 UTC m=+2610.643473411" Oct 30 06:15:04.117162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.117101 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" podStartSLOduration=4.117082765 podStartE2EDuration="4.117082765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.108727146 +0000 UTC m=+2610.684414200" watchObservedRunningTime="2026-10-30 06:15:04.117082765 +0000 UTC m=+2610.692769822" Oct 30 06:15:04.179411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.179319 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" podStartSLOduration=4.179299896 podStartE2EDuration="4.179299896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.152192128 +0000 UTC m=+2610.727879183" watchObservedRunningTime="2026-10-30 06:15:04.179299896 +0000 UTC m=+2610.754986951" Oct 30 06:15:04.191596 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.191443 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" podUID="81e69693-69c0-4d07-867e-8d385f41c342" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:15:04.192710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.192653 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:15:04.231137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.231075 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" podStartSLOduration=4.231055679 podStartE2EDuration="4.231055679s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.216097193 +0000 UTC m=+2610.791784248" watchObservedRunningTime="2026-10-30 06:15:04.231055679 +0000 UTC m=+2610.806742734" Oct 30 06:15:04.342225 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.339886 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" podStartSLOduration=4.339868088 podStartE2EDuration="4.339868088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:04.332418417 +0000 UTC m=+2610.908105472" watchObservedRunningTime="2026-10-30 06:15:04.339868088 +0000 UTC m=+2610.915555143" Oct 30 06:15:04.511642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.511613 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 06:15:04.619384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.619342 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.619734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.619702 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.619977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.619958 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.620141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.620122 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.620425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.620392 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.620642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.620609 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.620810 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.620793 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.621037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.621019 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.621375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.621356 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket\") pod \"67c3d184-a561-4c40-a950-6e93e7ae48de\" (UID: \"67c3d184-a561-4c40-a950-6e93e7ae48de\") " Oct 30 06:15:04.623045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.623022 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:04.624773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.624751 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:04.626840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.626818 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.626971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.626957 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.627404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.627387 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:04.627770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.627741 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:04.645621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.644877 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:04.710712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.710646 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:15:04.711586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.711519 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "kube-api-access-74j2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:04.715179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.713930 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:04.722634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.722342 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" start-of-body= Oct 30 06:15:04.722634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.722401 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.244:15021/healthz/ready\": dial tcp 10.134.0.244:15021: connect: connection refused" Oct 30 06:15:04.726249 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.726216 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy" pod "67c3d184-a561-4c40-a950-6e93e7ae48de" (UID: "67c3d184-a561-4c40-a950-6e93e7ae48de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729773 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729805 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729817 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729830 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74j2s\" (UniqueName: \"kubernetes.io/projected/67c3d184-a561-4c40-a950-6e93e7ae48de-kube-api-access-74j2s\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729854 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67c3d184-a561-4c40-a950-6e93e7ae48de-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729870 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67c3d184-a561-4c40-a950-6e93e7ae48de-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:04.730179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:04.729884 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67c3d184-a561-4c40-a950-6e93e7ae48de-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:05.230200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.230166 5565 generic.go:356] "Generic (PLEG): container finished" podID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerID="073ab0e1e907b7d218c7d64828f03947cdbfacf318545d65a25f6d3ae6d96c1f" exitCode=60 Oct 30 06:15:05.230571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.230504 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" event={"ID":"196a2b1b-619c-416b-a839-e3b1fd15699f","Type":"ContainerDied","Data":"073ab0e1e907b7d218c7d64828f03947cdbfacf318545d65a25f6d3ae6d96c1f"} Oct 30 06:15:05.231628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.231597 5565 scope.go:122] "RemoveContainer" containerID="073ab0e1e907b7d218c7d64828f03947cdbfacf318545d65a25f6d3ae6d96c1f" Oct 30 06:15:05.234021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.233986 5565 generic.go:356] "Generic (PLEG): container finished" podID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerID="77379c0c17f3fffc8decbb6ec7777a3df722a59bfdadfb7f4368144d16a60ac2" exitCode=60 Oct 30 06:15:05.234223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.234204 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerDied","Data":"77379c0c17f3fffc8decbb6ec7777a3df722a59bfdadfb7f4368144d16a60ac2"} Oct 30 06:15:05.235742 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.235723 5565 scope.go:122] "RemoveContainer" containerID="77379c0c17f3fffc8decbb6ec7777a3df722a59bfdadfb7f4368144d16a60ac2" Oct 30 06:15:05.239822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.237937 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" Oct 30 06:15:05.239822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.238509 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d" event={"ID":"67c3d184-a561-4c40-a950-6e93e7ae48de","Type":"ContainerDied","Data":"d1ee25e210a9b6125325c01d48165318b6c5376c2abfb584fca2d60707a29a19"} Oct 30 06:15:05.239822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.238578 5565 scope.go:122] "RemoveContainer" containerID="21ffeb60497120eb2906eff3a2fa46b4694f90cce39f1e19c6761d25eda0dfa1" Oct 30 06:15:05.247049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.247022 5565 generic.go:356] "Generic (PLEG): container finished" podID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerID="23800d1103a65b685cec4323fb9c98afa05ba393809eb7a8feb1e5b9375c0477" exitCode=60 Oct 30 06:15:05.247160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.247125 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerDied","Data":"23800d1103a65b685cec4323fb9c98afa05ba393809eb7a8feb1e5b9375c0477"} Oct 30 06:15:05.247597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.247499 5565 scope.go:122] "RemoveContainer" containerID="23800d1103a65b685cec4323fb9c98afa05ba393809eb7a8feb1e5b9375c0477" Oct 30 06:15:05.295196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.295163 5565 generic.go:356] "Generic (PLEG): container finished" podID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerID="3b72b0383882f7e16bc7bda56f8a70b2dd8dfdc6659fb2fdffb42320975b77dd" exitCode=60 Oct 30 06:15:05.295458 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.295438 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerDied","Data":"3b72b0383882f7e16bc7bda56f8a70b2dd8dfdc6659fb2fdffb42320975b77dd"} Oct 30 06:15:05.296334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.296301 5565 scope.go:122] "RemoveContainer" containerID="3b72b0383882f7e16bc7bda56f8a70b2dd8dfdc6659fb2fdffb42320975b77dd" Oct 30 06:15:05.308586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.308246 5565 generic.go:356] "Generic (PLEG): container finished" podID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerID="8bba5c65563d5a3336b2f39d4e15776e540fa506de385e7deb6a9f0edce19ef8" exitCode=60 Oct 30 06:15:05.308586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.308304 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" event={"ID":"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b","Type":"ContainerDied","Data":"8bba5c65563d5a3336b2f39d4e15776e540fa506de385e7deb6a9f0edce19ef8"} Oct 30 06:15:05.308723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.308644 5565 scope.go:122] "RemoveContainer" containerID="8bba5c65563d5a3336b2f39d4e15776e540fa506de385e7deb6a9f0edce19ef8" Oct 30 06:15:05.325739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.324745 5565 generic.go:356] "Generic (PLEG): container finished" podID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerID="5f2e26ca60358d42c180479aefe11cbc07e3dd5510a80652834df12e36ba1da2" exitCode=60 Oct 30 06:15:05.325739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.324820 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerDied","Data":"5f2e26ca60358d42c180479aefe11cbc07e3dd5510a80652834df12e36ba1da2"} Oct 30 06:15:05.328647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.328625 5565 generic.go:356] "Generic (PLEG): container finished" podID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerID="e98e6ec2ed41c1dedd1041653e9b931c24336349b47784473b9af9d4ea000200" exitCode=60 Oct 30 06:15:05.328760 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.328721 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerDied","Data":"e98e6ec2ed41c1dedd1041653e9b931c24336349b47784473b9af9d4ea000200"} Oct 30 06:15:05.329062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.328930 5565 scope.go:122] "RemoveContainer" containerID="e98e6ec2ed41c1dedd1041653e9b931c24336349b47784473b9af9d4ea000200" Oct 30 06:15:05.331429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.330931 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe912d29-01ab-4b03-a703-613a525154d1" containerID="4a9a37c62e214595b0bddb261f5c83600abefb34d68373a2d5a59c674d1b3090" exitCode=60 Oct 30 06:15:05.331429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.330980 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerDied","Data":"4a9a37c62e214595b0bddb261f5c83600abefb34d68373a2d5a59c674d1b3090"} Oct 30 06:15:05.332160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.331779 5565 scope.go:122] "RemoveContainer" containerID="4a9a37c62e214595b0bddb261f5c83600abefb34d68373a2d5a59c674d1b3090" Oct 30 06:15:05.333092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.333072 5565 generic.go:356] "Generic (PLEG): container finished" podID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerID="58519c6528312174eb3d57ba24a1c2671819be1bdfb3d506ca7a1f839dc00441" exitCode=60 Oct 30 06:15:05.333519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.333226 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerDied","Data":"58519c6528312174eb3d57ba24a1c2671819be1bdfb3d506ca7a1f839dc00441"} Oct 30 06:15:05.334215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.334199 5565 scope.go:122] "RemoveContainer" containerID="58519c6528312174eb3d57ba24a1c2671819be1bdfb3d506ca7a1f839dc00441" Oct 30 06:15:05.335623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.335367 5565 generic.go:356] "Generic (PLEG): container finished" podID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerID="8435c949bdded446d03e89cc3e02e9ed03455015424f2656cb0aec24fc6df6b4" exitCode=60 Oct 30 06:15:05.335623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.335498 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerDied","Data":"8435c949bdded446d03e89cc3e02e9ed03455015424f2656cb0aec24fc6df6b4"} Oct 30 06:15:05.336005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.335866 5565 scope.go:122] "RemoveContainer" containerID="8435c949bdded446d03e89cc3e02e9ed03455015424f2656cb0aec24fc6df6b4" Oct 30 06:15:05.345646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.345621 5565 generic.go:356] "Generic (PLEG): container finished" podID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerID="1109f8479f852e0c1353299daaf23ed5a829a536e77ddaa8f841ede5b04a2c40" exitCode=60 Oct 30 06:15:05.345736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.345664 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerDied","Data":"1109f8479f852e0c1353299daaf23ed5a829a536e77ddaa8f841ede5b04a2c40"} Oct 30 06:15:05.346000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.345965 5565 scope.go:122] "RemoveContainer" containerID="1109f8479f852e0c1353299daaf23ed5a829a536e77ddaa8f841ede5b04a2c40" Oct 30 06:15:05.841409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.841350 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" podStartSLOduration=5.841336159 podStartE2EDuration="5.841336159s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:05.838075293 +0000 UTC m=+2612.413762347" watchObservedRunningTime="2026-10-30 06:15:05.841336159 +0000 UTC m=+2612.417023246" Oct 30 06:15:05.844761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.844730 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:15:05.845200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.845178 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:15:05.846107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.846082 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:15:05.906405 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.906317 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d"] Oct 30 06:15:05.911071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.911045 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-0b481b-openshift-default-6b4bc57457-jkl6d"] Oct 30 06:15:05.947147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:05.947085 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" podStartSLOduration=5.947068324 podStartE2EDuration="5.947068324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:05.945110302 +0000 UTC m=+2612.520797357" watchObservedRunningTime="2026-10-30 06:15:05.947068324 +0000 UTC m=+2612.522755379" Oct 30 06:15:06.362007 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.360222 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" event={"ID":"196a2b1b-619c-416b-a839-e3b1fd15699f","Type":"ContainerStarted","Data":"6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad"} Oct 30 06:15:06.363014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.362866 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerStarted","Data":"3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e"} Oct 30 06:15:06.368465 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.368351 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerStarted","Data":"de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c"} Oct 30 06:15:06.370727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.370702 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerStarted","Data":"83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4"} Oct 30 06:15:06.373567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.373060 5565 generic.go:356] "Generic (PLEG): container finished" podID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerID="a82741732898b06fd5469cb199b89bebcaeff60159adb15d4c47ab06d825ab9d" exitCode=60 Oct 30 06:15:06.373567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.373128 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerDied","Data":"a82741732898b06fd5469cb199b89bebcaeff60159adb15d4c47ab06d825ab9d"} Oct 30 06:15:06.373567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.373171 5565 scope.go:122] "RemoveContainer" containerID="5f2e26ca60358d42c180479aefe11cbc07e3dd5510a80652834df12e36ba1da2" Oct 30 06:15:06.373567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.373387 5565 scope.go:122] "RemoveContainer" containerID="a82741732898b06fd5469cb199b89bebcaeff60159adb15d4c47ab06d825ab9d" Oct 30 06:15:06.373837 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:06.373666 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p_odh-ai-gateway-infra(62d68114-a3f3-48eb-b3ca-ff1bdbbd6504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" Oct 30 06:15:06.376184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.375431 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerStarted","Data":"41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116"} Oct 30 06:15:06.378274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.378249 5565 generic.go:356] "Generic (PLEG): container finished" podID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerID="41d2ef7695bffd00524780bf536f6ea5dd56be4fe902afc2d0ce91248782748c" exitCode=60 Oct 30 06:15:06.378362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.378321 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" event={"ID":"633df5e9-b8cf-480a-9b49-1a013f55e729","Type":"ContainerDied","Data":"41d2ef7695bffd00524780bf536f6ea5dd56be4fe902afc2d0ce91248782748c"} Oct 30 06:15:06.378707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.378684 5565 scope.go:122] "RemoveContainer" containerID="41d2ef7695bffd00524780bf536f6ea5dd56be4fe902afc2d0ce91248782748c" Oct 30 06:15:06.379037 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:06.379010 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb_odh-ai-gateway-infra(633df5e9-b8cf-480a-9b49-1a013f55e729)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" Oct 30 06:15:06.381649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.381620 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerStarted","Data":"9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186"} Oct 30 06:15:06.391285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.391254 5565 scope.go:122] "RemoveContainer" containerID="599f6f23ec01fa925c7ea9ef09c41a08dd23872361b08399a005c471c4a540b0" Oct 30 06:15:06.412212 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.412149 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" podStartSLOduration=6.412135247 podStartE2EDuration="6.412135247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:06.411336503 +0000 UTC m=+2612.987023630" watchObservedRunningTime="2026-10-30 06:15:06.412135247 +0000 UTC m=+2612.987822300" Oct 30 06:15:06.592920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.592862 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" podStartSLOduration=6.592849283 podStartE2EDuration="6.592849283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:06.590214067 +0000 UTC m=+2613.165901136" watchObservedRunningTime="2026-10-30 06:15:06.592849283 +0000 UTC m=+2613.168536371" Oct 30 06:15:06.819554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.813186 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4"] Oct 30 06:15:06.819554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.814642 5565 cpu_manager.go:414] "containerMap: removing container" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" Oct 30 06:15:06.819554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.814666 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" Oct 30 06:15:06.819554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.815032 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" containerName="istio-proxy" Oct 30 06:15:06.879891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.865137 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:06.879891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.873683 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4"] Oct 30 06:15:06.910639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.900836 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-de2bf2\"" Oct 30 06:15:06.910639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.900980 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-de2bf2-dockercfg-nkbp2\"" Oct 30 06:15:06.913807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.913751 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" podStartSLOduration=6.913733413 podStartE2EDuration="6.913733413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:06.90525401 +0000 UTC m=+2613.480941302" watchObservedRunningTime="2026-10-30 06:15:06.913733413 +0000 UTC m=+2613.489420467" Oct 30 06:15:06.971003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.970966 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g49z4\" (UniqueName: \"kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:06.971218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:06.971197 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:07.072992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.072963 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:07.073152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.073134 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g49z4\" (UniqueName: \"kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:07.073740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.073716 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:07.098917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.098894 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g49z4\" (UniqueName: \"kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:07.392576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.392515 5565 scope.go:122] "RemoveContainer" containerID="a82741732898b06fd5469cb199b89bebcaeff60159adb15d4c47ab06d825ab9d" Oct 30 06:15:07.392941 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.392913 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p_odh-ai-gateway-infra(62d68114-a3f3-48eb-b3ca-ff1bdbbd6504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" Oct 30 06:15:07.394294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.394269 5565 generic.go:356] "Generic (PLEG): container finished" podID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerID="41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116" exitCode=60 Oct 30 06:15:07.394413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.394337 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerDied","Data":"41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116"} Oct 30 06:15:07.394413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.394394 5565 scope.go:122] "RemoveContainer" containerID="e98e6ec2ed41c1dedd1041653e9b931c24336349b47784473b9af9d4ea000200" Oct 30 06:15:07.394681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.394657 5565 scope.go:122] "RemoveContainer" containerID="41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116" Oct 30 06:15:07.394994 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.394970 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r_odh-ai-gateway-infra(7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" Oct 30 06:15:07.396678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.396646 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe912d29-01ab-4b03-a703-613a525154d1" containerID="e5f5b6d4210bad7c4c602f756a5b2e14447614e11da5b07f364bdb1a1db77d4b" exitCode=60 Oct 30 06:15:07.396774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.396722 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerDied","Data":"e5f5b6d4210bad7c4c602f756a5b2e14447614e11da5b07f364bdb1a1db77d4b"} Oct 30 06:15:07.397016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.396992 5565 scope.go:122] "RemoveContainer" containerID="e5f5b6d4210bad7c4c602f756a5b2e14447614e11da5b07f364bdb1a1db77d4b" Oct 30 06:15:07.397250 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.397232 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg_odh-ai-gateway-infra(fe912d29-01ab-4b03-a703-613a525154d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" podUID="fe912d29-01ab-4b03-a703-613a525154d1" Oct 30 06:15:07.398974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.398953 5565 generic.go:356] "Generic (PLEG): container finished" podID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerID="08cace4dff2346d5729a31a3cac60ca8202482357b1f8ffc43b97ec7460479c1" exitCode=60 Oct 30 06:15:07.399085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.399007 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerDied","Data":"08cace4dff2346d5729a31a3cac60ca8202482357b1f8ffc43b97ec7460479c1"} Oct 30 06:15:07.399195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.399179 5565 scope.go:122] "RemoveContainer" containerID="08cace4dff2346d5729a31a3cac60ca8202482357b1f8ffc43b97ec7460479c1" Oct 30 06:15:07.399411 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.399392 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6_odh-ai-gateway-infra(af0c5f69-23fd-4032-a17d-e41ee3336e36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" Oct 30 06:15:07.401139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.401104 5565 generic.go:356] "Generic (PLEG): container finished" podID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerID="c80529296f4a639dcc7aeaaf9190655c982c643a5c50bddbf34312f807644414" exitCode=60 Oct 30 06:15:07.401238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.401178 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerDied","Data":"c80529296f4a639dcc7aeaaf9190655c982c643a5c50bddbf34312f807644414"} Oct 30 06:15:07.401387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.401372 5565 scope.go:122] "RemoveContainer" containerID="c80529296f4a639dcc7aeaaf9190655c982c643a5c50bddbf34312f807644414" Oct 30 06:15:07.401694 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.401663 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg_odh-ai-gateway-infra(fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" Oct 30 06:15:07.405074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.405044 5565 generic.go:356] "Generic (PLEG): container finished" podID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerID="9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186" exitCode=60 Oct 30 06:15:07.405178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.405128 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerDied","Data":"9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186"} Oct 30 06:15:07.405337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.405321 5565 scope.go:122] "RemoveContainer" containerID="9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186" Oct 30 06:15:07.405591 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.405523 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk_odh-ai-gateway-infra(ba63fa73-dbd3-466a-b288-9ed56a958e80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" Oct 30 06:15:07.407663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.407641 5565 generic.go:356] "Generic (PLEG): container finished" podID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerID="6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad" exitCode=60 Oct 30 06:15:07.407807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.407756 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" event={"ID":"196a2b1b-619c-416b-a839-e3b1fd15699f","Type":"ContainerDied","Data":"6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad"} Oct 30 06:15:07.413186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.408457 5565 scope.go:122] "RemoveContainer" containerID="6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad" Oct 30 06:15:07.413186 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.408926 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf_odh-ai-gateway-infra(196a2b1b-619c-416b-a839-e3b1fd15699f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.429687 5565 generic.go:356] "Generic (PLEG): container finished" podID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerID="3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e" exitCode=60 Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.429885 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerDied","Data":"3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e"} Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.430390 5565 scope.go:122] "RemoveContainer" containerID="3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e" Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.430884 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl_odh-ai-gateway-infra(cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.453918 5565 generic.go:356] "Generic (PLEG): container finished" podID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerID="de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c" exitCode=60 Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.454032 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerDied","Data":"de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c"} Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.455196 5565 scope.go:122] "RemoveContainer" containerID="de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c" Oct 30 06:15:07.458794 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.455841 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b_odh-ai-gateway-infra(3f2c5ea0-81b3-48c7-942e-7c064f1871bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" Oct 30 06:15:07.459665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.459607 5565 scope.go:122] "RemoveContainer" containerID="4a9a37c62e214595b0bddb261f5c83600abefb34d68373a2d5a59c674d1b3090" Oct 30 06:15:07.463585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.463523 5565 generic.go:356] "Generic (PLEG): container finished" podID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerID="83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4" exitCode=60 Oct 30 06:15:07.463708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.463667 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerDied","Data":"83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4"} Oct 30 06:15:07.463890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.463862 5565 scope.go:122] "RemoveContainer" containerID="83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4" Oct 30 06:15:07.464310 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.464285 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f_odh-ai-gateway-infra(cbe50e49-14ed-4ad1-8ca6-213effbd0a45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" Oct 30 06:15:07.466230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.466205 5565 generic.go:356] "Generic (PLEG): container finished" podID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerID="14963ce2113066a12fecc6f87702a8fe1f6ade43f16950de420233332445a325" exitCode=60 Oct 30 06:15:07.466333 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.466291 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" event={"ID":"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b","Type":"ContainerDied","Data":"14963ce2113066a12fecc6f87702a8fe1f6ade43f16950de420233332445a325"} Oct 30 06:15:07.466665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.466646 5565 scope.go:122] "RemoveContainer" containerID="14963ce2113066a12fecc6f87702a8fe1f6ade43f16950de420233332445a325" Oct 30 06:15:07.466936 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:07.466915 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv_odh-ai-gateway-infra(9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" Oct 30 06:15:07.470047 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.470027 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:07.477368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.477351 5565 scope.go:122] "RemoveContainer" containerID="58519c6528312174eb3d57ba24a1c2671819be1bdfb3d506ca7a1f839dc00441" Oct 30 06:15:07.500410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.500390 5565 scope.go:122] "RemoveContainer" containerID="8435c949bdded446d03e89cc3e02e9ed03455015424f2656cb0aec24fc6df6b4" Oct 30 06:15:07.515181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.515156 5565 scope.go:122] "RemoveContainer" containerID="1109f8479f852e0c1353299daaf23ed5a829a536e77ddaa8f841ede5b04a2c40" Oct 30 06:15:07.533186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.533164 5565 scope.go:122] "RemoveContainer" containerID="073ab0e1e907b7d218c7d64828f03947cdbfacf318545d65a25f6d3ae6d96c1f" Oct 30 06:15:07.555700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.555574 5565 scope.go:122] "RemoveContainer" containerID="77379c0c17f3fffc8decbb6ec7777a3df722a59bfdadfb7f4368144d16a60ac2" Oct 30 06:15:07.584921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.584866 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67c3d184-a561-4c40-a950-6e93e7ae48de" path="/var/lib/kubelet/pods/67c3d184-a561-4c40-a950-6e93e7ae48de/volumes" Oct 30 06:15:07.596934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.596908 5565 scope.go:122] "RemoveContainer" containerID="23800d1103a65b685cec4323fb9c98afa05ba393809eb7a8feb1e5b9375c0477" Oct 30 06:15:07.623847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.623825 5565 scope.go:122] "RemoveContainer" containerID="3b72b0383882f7e16bc7bda56f8a70b2dd8dfdc6659fb2fdffb42320975b77dd" Oct 30 06:15:07.643809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.643786 5565 scope.go:122] "RemoveContainer" containerID="8bba5c65563d5a3336b2f39d4e15776e540fa506de385e7deb6a9f0edce19ef8" Oct 30 06:15:07.797595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:07.797567 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4"] Oct 30 06:15:07.797946 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:15:07.797925 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc92ace7_b66c_41c0_95e7_736a244b421f.slice/crio-3c6db571836387d5293c4577ca70bf293fb97664d36a0b40d9c0cd9bbd4e8901 WatchSource:0}: Error finding container 3c6db571836387d5293c4577ca70bf293fb97664d36a0b40d9c0cd9bbd4e8901: Status 404 returned error can't find the container with id 3c6db571836387d5293c4577ca70bf293fb97664d36a0b40d9c0cd9bbd4e8901 Oct 30 06:15:08.477397 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.477356 5565 scope.go:122] "RemoveContainer" containerID="6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad" Oct 30 06:15:08.477827 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.477789 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf_odh-ai-gateway-infra(196a2b1b-619c-416b-a839-e3b1fd15699f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" Oct 30 06:15:08.479463 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.479437 5565 scope.go:122] "RemoveContainer" containerID="3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e" Oct 30 06:15:08.479876 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.479849 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl_odh-ai-gateway-infra(cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" Oct 30 06:15:08.481395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.481375 5565 scope.go:122] "RemoveContainer" containerID="de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c" Oct 30 06:15:08.481829 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.481737 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b_odh-ai-gateway-infra(3f2c5ea0-81b3-48c7-942e-7c064f1871bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" Oct 30 06:15:08.483234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.483213 5565 scope.go:122] "RemoveContainer" containerID="83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4" Oct 30 06:15:08.483607 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.483582 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f_odh-ai-gateway-infra(cbe50e49-14ed-4ad1-8ca6-213effbd0a45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" Oct 30 06:15:08.485192 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.485088 5565 scope.go:122] "RemoveContainer" containerID="14963ce2113066a12fecc6f87702a8fe1f6ade43f16950de420233332445a325" Oct 30 06:15:08.485416 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.485395 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv_odh-ai-gateway-infra(9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" Oct 30 06:15:08.486847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.486827 5565 scope.go:122] "RemoveContainer" containerID="41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116" Oct 30 06:15:08.487127 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.487101 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r_odh-ai-gateway-infra(7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" Oct 30 06:15:08.488643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.488621 5565 scope.go:122] "RemoveContainer" containerID="e5f5b6d4210bad7c4c602f756a5b2e14447614e11da5b07f364bdb1a1db77d4b" Oct 30 06:15:08.488891 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.488868 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg_odh-ai-gateway-infra(fe912d29-01ab-4b03-a703-613a525154d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" podUID="fe912d29-01ab-4b03-a703-613a525154d1" Oct 30 06:15:08.490388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.490369 5565 scope.go:122] "RemoveContainer" containerID="08cace4dff2346d5729a31a3cac60ca8202482357b1f8ffc43b97ec7460479c1" Oct 30 06:15:08.490725 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.490700 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6_odh-ai-gateway-infra(af0c5f69-23fd-4032-a17d-e41ee3336e36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" Oct 30 06:15:08.492881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.492451 5565 scope.go:122] "RemoveContainer" containerID="c80529296f4a639dcc7aeaaf9190655c982c643a5c50bddbf34312f807644414" Oct 30 06:15:08.492881 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.492748 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg_odh-ai-gateway-infra(fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" Oct 30 06:15:08.494026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.493998 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" event={"ID":"cc92ace7-b66c-41c0-95e7-736a244b421f","Type":"ContainerStarted","Data":"c9d800761dc8083b9aac83b600febaccb15bf113ac57552704f86b97bad5ca0b"} Oct 30 06:15:08.494129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.494052 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" event={"ID":"cc92ace7-b66c-41c0-95e7-736a244b421f","Type":"ContainerStarted","Data":"3c6db571836387d5293c4577ca70bf293fb97664d36a0b40d9c0cd9bbd4e8901"} Oct 30 06:15:08.494129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.494112 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:08.496287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.496271 5565 scope.go:122] "RemoveContainer" containerID="9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186" Oct 30 06:15:08.496497 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:08.496480 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk_odh-ai-gateway-infra(ba63fa73-dbd3-466a-b288-9ed56a958e80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" Oct 30 06:15:08.704519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:08.704446 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" podStartSLOduration=2.704429264 podStartE2EDuration="2.704429264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:08.701982837 +0000 UTC m=+2615.277669892" watchObservedRunningTime="2026-10-30 06:15:08.704429264 +0000 UTC m=+2615.280116318" Oct 30 06:15:09.415496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:09.415461 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47"] Oct 30 06:15:09.444104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:09.444072 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl"] Oct 30 06:15:12.538341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:12.538308 5565 generic.go:356] "Generic (PLEG): container finished" podID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerID="46a0d14d45a7a0a93bbc4ef99a99f38b0be144615d59506eed0d9d2b38504ede" exitCode=0 Oct 30 06:15:12.538817 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:12.538378 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" event={"ID":"d078081e-660d-4c3a-8c14-dd61d34f9e82","Type":"ContainerDied","Data":"46a0d14d45a7a0a93bbc4ef99a99f38b0be144615d59506eed0d9d2b38504ede"} Oct 30 06:15:12.538907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:12.538891 5565 scope.go:122] "RemoveContainer" containerID="46a0d14d45a7a0a93bbc4ef99a99f38b0be144615d59506eed0d9d2b38504ede" Oct 30 06:15:12.540817 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:12.540790 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:15:12.540904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:12.540858 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:15:12.540942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:12.540910 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:15:13.087604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.087573 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 06:15:13.108885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.108861 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 06:15:13.172173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.172118 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume\") pod \"9fda9767-1e03-4e98-800f-e9ed34dedf47\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " Oct 30 06:15:13.172374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.172356 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w\" (UniqueName: \"kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w\") pod \"9fda9767-1e03-4e98-800f-e9ed34dedf47\" (UID: \"9fda9767-1e03-4e98-800f-e9ed34dedf47\") " Oct 30 06:15:13.172481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.172462 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z\" (UniqueName: \"kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z\") pod \"81e69693-69c0-4d07-867e-8d385f41c342\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " Oct 30 06:15:13.172606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.172550 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume" pod "9fda9767-1e03-4e98-800f-e9ed34dedf47" (UID: "9fda9767-1e03-4e98-800f-e9ed34dedf47"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:13.173054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.173029 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume\") pod \"81e69693-69c0-4d07-867e-8d385f41c342\" (UID: \"81e69693-69c0-4d07-867e-8d385f41c342\") " Oct 30 06:15:13.173651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.173522 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume" pod "81e69693-69c0-4d07-867e-8d385f41c342" (UID: "81e69693-69c0-4d07-867e-8d385f41c342"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:13.173892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.173871 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81e69693-69c0-4d07-867e-8d385f41c342-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.173974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.173897 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9fda9767-1e03-4e98-800f-e9ed34dedf47-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.174899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.174875 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w" pod "9fda9767-1e03-4e98-800f-e9ed34dedf47" (UID: "9fda9767-1e03-4e98-800f-e9ed34dedf47"). InnerVolumeSpecName "kube-api-access-qxg9w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:13.175136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.175112 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z" pod "81e69693-69c0-4d07-867e-8d385f41c342" (UID: "81e69693-69c0-4d07-867e-8d385f41c342"). InnerVolumeSpecName "kube-api-access-hsm5z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:13.274899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.274866 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxg9w\" (UniqueName: \"kubernetes.io/projected/9fda9767-1e03-4e98-800f-e9ed34dedf47-kube-api-access-qxg9w\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.274899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.274897 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsm5z\" (UniqueName: \"kubernetes.io/projected/81e69693-69c0-4d07-867e-8d385f41c342-kube-api-access-hsm5z\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:13.547766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.547732 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" event={"ID":"d078081e-660d-4c3a-8c14-dd61d34f9e82","Type":"ContainerStarted","Data":"4e74a78ce719d1c75aa00fd323644ca349319f1bc8c4f0e6cdd0a492d23b9c0d"} Oct 30 06:15:13.549153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.549127 5565 generic.go:356] "Generic (PLEG): container finished" podID="81e69693-69c0-4d07-867e-8d385f41c342" containerID="434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791" exitCode=0 Oct 30 06:15:13.549283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.549175 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" Oct 30 06:15:13.549283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.549204 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" event={"ID":"81e69693-69c0-4d07-867e-8d385f41c342","Type":"ContainerDied","Data":"434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791"} Oct 30 06:15:13.549283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.549244 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw" event={"ID":"81e69693-69c0-4d07-867e-8d385f41c342","Type":"ContainerDied","Data":"c4aa5c1e3202d8deb167eb5d82a11eadfc7b07f9ac8175dfa920abdf785bb528"} Oct 30 06:15:13.549283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.549265 5565 scope.go:122] "RemoveContainer" containerID="434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791" Oct 30 06:15:13.550657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.550632 5565 generic.go:356] "Generic (PLEG): container finished" podID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerID="566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25" exitCode=0 Oct 30 06:15:13.550762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.550679 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" event={"ID":"9fda9767-1e03-4e98-800f-e9ed34dedf47","Type":"ContainerDied","Data":"566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25"} Oct 30 06:15:13.550762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.550685 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" Oct 30 06:15:13.550762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.550716 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7" event={"ID":"9fda9767-1e03-4e98-800f-e9ed34dedf47","Type":"ContainerDied","Data":"2a0ac6f8783e3b07ebeebe7e1b125fc7b730c1110534e64e59d90663fc506e7b"} Oct 30 06:15:13.561811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.561792 5565 scope.go:122] "RemoveContainer" containerID="434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791" Oct 30 06:15:13.562113 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:13.562085 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791\": container with ID starting with 434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791 not found: ID does not exist" containerID="434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791" Oct 30 06:15:13.562196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.562122 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791"} err="failed to get container status \"434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791\": rpc error: code = NotFound desc = could not find container \"434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791\": container with ID starting with 434d1b3cfb7e5d30d71302f1c28cf003c6c6cc7c8fba577b32a9828df5b71791 not found: ID does not exist" Oct 30 06:15:13.562196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.562146 5565 scope.go:122] "RemoveContainer" containerID="566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25" Oct 30 06:15:13.574064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.574049 5565 scope.go:122] "RemoveContainer" containerID="566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25" Oct 30 06:15:13.574308 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:13.574287 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25\": container with ID starting with 566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25 not found: ID does not exist" containerID="566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25" Oct 30 06:15:13.574381 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.574317 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25"} err="failed to get container status \"566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25\": rpc error: code = NotFound desc = could not find container \"566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25\": container with ID starting with 566e382772a9a2c06e8fe1d7ed5af789b2ea0f5b1409d250b3d57fc1e6c2ac25 not found: ID does not exist" Oct 30 06:15:13.644518 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.644492 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7"] Oct 30 06:15:13.653854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.653829 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-0b481b-84b4f468db-fpzz7"] Oct 30 06:15:13.712294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.712263 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw"] Oct 30 06:15:13.720001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:13.719977 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-0b481b-55fd7fb787-bg4bw"] Oct 30 06:15:14.207557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:14.203934 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:15:14.207557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:14.204069 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:15:14.207557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:14.204115 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 06:15:14.415777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:14.415735 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerName="payload-processing" containerID="cri-o://9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198" gracePeriod=25 Oct 30 06:15:14.444935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:14.444896 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerName="payload-pre-processing" containerID="cri-o://4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315" gracePeriod=25 Oct 30 06:15:15.194369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:15.194288 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:15:15.194369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:15.194354 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 06:15:15.586753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:15.586727 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81e69693-69c0-4d07-867e-8d385f41c342" path="/var/lib/kubelet/pods/81e69693-69c0-4d07-867e-8d385f41c342/volumes" Oct 30 06:15:15.587353 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:15.587332 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" path="/var/lib/kubelet/pods/9fda9767-1e03-4e98-800f-e9ed34dedf47/volumes" Oct 30 06:15:16.194654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:16.194615 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:15:16.195070 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:16.194672 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 06:15:17.193986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:17.193949 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:15:17.194145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:17.194014 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 06:15:17.535121 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:17.535086 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:15:17.566550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:17.563032 5565 scope.go:122] "RemoveContainer" containerID="a82741732898b06fd5469cb199b89bebcaeff60159adb15d4c47ab06d825ab9d" Oct 30 06:15:17.566550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:17.564652 5565 scope.go:122] "RemoveContainer" containerID="e5f5b6d4210bad7c4c602f756a5b2e14447614e11da5b07f364bdb1a1db77d4b" Oct 30 06:15:17.566550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:17.565316 5565 scope.go:122] "RemoveContainer" containerID="83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4" Oct 30 06:15:18.194149 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.194117 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:15:18.194326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.194175 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 06:15:18.581019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.575126 5565 scope.go:122] "RemoveContainer" containerID="9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186" Oct 30 06:15:18.581019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.576093 5565 scope.go:122] "RemoveContainer" containerID="de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c" Oct 30 06:15:18.621942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.621912 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerStarted","Data":"9e2973db3588b8e0947a27682795a2c6c1cd0e08114824efa584eeef53f8170d"} Oct 30 06:15:18.624888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.624862 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerStarted","Data":"5bffb1aa32c3f9e5ffb3b4bd503baf2860b254fc770ccac77c055b0ed7ce08b7"} Oct 30 06:15:18.627706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:18.627680 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerStarted","Data":"bed9d6a4e5a2b19b956d579c245b5fca2781f7fe7c156062c6c47e6c5caef4ca"} Oct 30 06:15:19.194146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.194067 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" start-of-body= Oct 30 06:15:19.194146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.194119 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.237:15021/healthz/ready\": dial tcp 10.134.0.237:15021: connect: connection refused" Oct 30 06:15:19.519654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.519625 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:15:19.560845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.560810 5565 scope.go:122] "RemoveContainer" containerID="08cace4dff2346d5729a31a3cac60ca8202482357b1f8ffc43b97ec7460479c1" Oct 30 06:15:19.562127 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.562101 5565 scope.go:122] "RemoveContainer" containerID="41d2ef7695bffd00524780bf536f6ea5dd56be4fe902afc2d0ce91248782748c" Oct 30 06:15:19.563479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.562464 5565 scope.go:122] "RemoveContainer" containerID="14963ce2113066a12fecc6f87702a8fe1f6ade43f16950de420233332445a325" Oct 30 06:15:19.646598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.646568 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerStarted","Data":"3a642a854de7ed59d95c6b241d28e64a8fdb07a805c2f91d61b8ea971b628af0"} Oct 30 06:15:19.648559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.648505 5565 generic.go:356] "Generic (PLEG): container finished" podID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerID="9e2973db3588b8e0947a27682795a2c6c1cd0e08114824efa584eeef53f8170d" exitCode=60 Oct 30 06:15:19.648654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.648604 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerDied","Data":"9e2973db3588b8e0947a27682795a2c6c1cd0e08114824efa584eeef53f8170d"} Oct 30 06:15:19.648654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.648647 5565 scope.go:122] "RemoveContainer" containerID="83840ddaeffe594dda94346725aed1bc2590dd1c1a28f79c9d2a14637da7bac4" Oct 30 06:15:19.648988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.648974 5565 scope.go:122] "RemoveContainer" containerID="9e2973db3588b8e0947a27682795a2c6c1cd0e08114824efa584eeef53f8170d" Oct 30 06:15:19.649230 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:19.649213 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f_odh-ai-gateway-infra(cbe50e49-14ed-4ad1-8ca6-213effbd0a45)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" Oct 30 06:15:19.651377 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.651358 5565 generic.go:356] "Generic (PLEG): container finished" podID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerID="5bffb1aa32c3f9e5ffb3b4bd503baf2860b254fc770ccac77c055b0ed7ce08b7" exitCode=60 Oct 30 06:15:19.651452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.651428 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerDied","Data":"5bffb1aa32c3f9e5ffb3b4bd503baf2860b254fc770ccac77c055b0ed7ce08b7"} Oct 30 06:15:19.652072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.651799 5565 scope.go:122] "RemoveContainer" containerID="5bffb1aa32c3f9e5ffb3b4bd503baf2860b254fc770ccac77c055b0ed7ce08b7" Oct 30 06:15:19.652312 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:19.652286 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p_odh-ai-gateway-infra(62d68114-a3f3-48eb-b3ca-ff1bdbbd6504)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" Oct 30 06:15:19.653850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.653825 5565 generic.go:356] "Generic (PLEG): container finished" podID="fe912d29-01ab-4b03-a703-613a525154d1" containerID="bed9d6a4e5a2b19b956d579c245b5fca2781f7fe7c156062c6c47e6c5caef4ca" exitCode=60 Oct 30 06:15:19.653952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.653891 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerDied","Data":"bed9d6a4e5a2b19b956d579c245b5fca2781f7fe7c156062c6c47e6c5caef4ca"} Oct 30 06:15:19.654169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.654154 5565 scope.go:122] "RemoveContainer" containerID="bed9d6a4e5a2b19b956d579c245b5fca2781f7fe7c156062c6c47e6c5caef4ca" Oct 30 06:15:19.654380 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:19.654363 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg_odh-ai-gateway-infra(fe912d29-01ab-4b03-a703-613a525154d1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" podUID="fe912d29-01ab-4b03-a703-613a525154d1" Oct 30 06:15:19.656537 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.656502 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerStarted","Data":"36315b08ba8ca6353498511c85b1d8d5b46d181e5e46baaefde06c50b5811d26"} Oct 30 06:15:19.684319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.684270 5565 scope.go:122] "RemoveContainer" containerID="a82741732898b06fd5469cb199b89bebcaeff60159adb15d4c47ab06d825ab9d" Oct 30 06:15:19.691908 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.691878 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f"] Oct 30 06:15:19.703634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.703614 5565 scope.go:122] "RemoveContainer" containerID="e5f5b6d4210bad7c4c602f756a5b2e14447614e11da5b07f364bdb1a1db77d4b" Oct 30 06:15:19.752005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.751979 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg"] Oct 30 06:15:19.827285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:19.827221 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p"] Oct 30 06:15:20.199296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.199268 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:15:20.199855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.199834 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:15:20.200692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.200670 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:15:20.560177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.560144 5565 scope.go:122] "RemoveContainer" containerID="c80529296f4a639dcc7aeaaf9190655c982c643a5c50bddbf34312f807644414" Oct 30 06:15:20.560392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.560229 5565 scope.go:122] "RemoveContainer" containerID="41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116" Oct 30 06:15:20.676592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.676560 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerStarted","Data":"b9377487081e7cdacabf206b59c5a76f7daa8a9a717a45b860068198ff1b41a4"} Oct 30 06:15:20.678903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.678882 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" event={"ID":"633df5e9-b8cf-480a-9b49-1a013f55e729","Type":"ContainerStarted","Data":"1a73a907514a83174ec5fc8e4253168f4bd75067067de0a0a897a08f80c22a87"} Oct 30 06:15:20.680821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.680801 5565 generic.go:356] "Generic (PLEG): container finished" podID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerID="36315b08ba8ca6353498511c85b1d8d5b46d181e5e46baaefde06c50b5811d26" exitCode=60 Oct 30 06:15:20.680903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.680870 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerDied","Data":"36315b08ba8ca6353498511c85b1d8d5b46d181e5e46baaefde06c50b5811d26"} Oct 30 06:15:20.680954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.680922 5565 scope.go:122] "RemoveContainer" containerID="9f30bb9af0873380504297346bb8f46f99b7764454fac41d42f78dfae6d9c186" Oct 30 06:15:20.681240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.681223 5565 scope.go:122] "RemoveContainer" containerID="36315b08ba8ca6353498511c85b1d8d5b46d181e5e46baaefde06c50b5811d26" Oct 30 06:15:20.681585 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:20.681520 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk_odh-ai-gateway-infra(ba63fa73-dbd3-466a-b288-9ed56a958e80)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" Oct 30 06:15:20.683008 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.682991 5565 generic.go:356] "Generic (PLEG): container finished" podID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerID="3a642a854de7ed59d95c6b241d28e64a8fdb07a805c2f91d61b8ea971b628af0" exitCode=60 Oct 30 06:15:20.683096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.683061 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerDied","Data":"3a642a854de7ed59d95c6b241d28e64a8fdb07a805c2f91d61b8ea971b628af0"} Oct 30 06:15:20.683386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.683370 5565 scope.go:122] "RemoveContainer" containerID="3a642a854de7ed59d95c6b241d28e64a8fdb07a805c2f91d61b8ea971b628af0" Oct 30 06:15:20.683634 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:20.683615 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b_odh-ai-gateway-infra(3f2c5ea0-81b3-48c7-942e-7c064f1871bf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" Oct 30 06:15:20.686513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.686493 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" event={"ID":"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b","Type":"ContainerStarted","Data":"a20aa8808c55a1fc0d662336e0b7ac7ce5227f763f7559bc95ccde09440a17ba"} Oct 30 06:15:20.696503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.696486 5565 scope.go:122] "RemoveContainer" containerID="de83f081438b5b17869da9698804396765ed44b32118b5cce6bb8ef3d350cb5c" Oct 30 06:15:20.714153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.714133 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b"] Oct 30 06:15:20.876976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.876950 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk"] Oct 30 06:15:20.956120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:20.956061 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" podStartSLOduration=20.956042506 podStartE2EDuration="20.956042506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:20.951883057 +0000 UTC m=+2627.527570108" watchObservedRunningTime="2026-10-30 06:15:20.956042506 +0000 UTC m=+2627.531729560" Oct 30 06:15:21.065336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.065304 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:15:21.065743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.065718 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:15:21.091554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.091478 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:21.095972 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.095948 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:21.099872 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.099854 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:21.273341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.273305 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l\" (UniqueName: \"kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l\") pod \"fe912d29-01ab-4b03-a703-613a525154d1\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " Oct 30 06:15:21.273562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.273413 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca\") pod \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " Oct 30 06:15:21.273562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.273445 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca\") pod \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " Oct 30 06:15:21.273562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.273483 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc\" (UniqueName: \"kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc\") pod \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\" (UID: \"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504\") " Oct 30 06:15:21.273766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.273715 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca\") pod \"fe912d29-01ab-4b03-a703-613a525154d1\" (UID: \"fe912d29-01ab-4b03-a703-613a525154d1\") " Oct 30 06:15:21.273766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.273751 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf\" (UniqueName: \"kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf\") pod \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\" (UID: \"cbe50e49-14ed-4ad1-8ca6-213effbd0a45\") " Oct 30 06:15:21.274054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.274018 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca" pod "cbe50e49-14ed-4ad1-8ca6-213effbd0a45" (UID: "cbe50e49-14ed-4ad1-8ca6-213effbd0a45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.274148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.274061 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca" pod "62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" (UID: "62d68114-a3f3-48eb-b3ca-ff1bdbbd6504"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.274437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.274410 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca" pod "fe912d29-01ab-4b03-a703-613a525154d1" (UID: "fe912d29-01ab-4b03-a703-613a525154d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:21.274931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.274904 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe912d29-01ab-4b03-a703-613a525154d1-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.275042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.274935 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.275042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.274950 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.276987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.276961 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf" pod "cbe50e49-14ed-4ad1-8ca6-213effbd0a45" (UID: "cbe50e49-14ed-4ad1-8ca6-213effbd0a45"). InnerVolumeSpecName "kube-api-access-t6hsf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.276987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.276984 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l" pod "fe912d29-01ab-4b03-a703-613a525154d1" (UID: "fe912d29-01ab-4b03-a703-613a525154d1"). InnerVolumeSpecName "kube-api-access-2mp9l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.277158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.276965 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc" pod "62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" (UID: "62d68114-a3f3-48eb-b3ca-ff1bdbbd6504"). InnerVolumeSpecName "kube-api-access-gd6zc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:21.376031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.375951 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6hsf\" (UniqueName: \"kubernetes.io/projected/cbe50e49-14ed-4ad1-8ca6-213effbd0a45-kube-api-access-t6hsf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.376031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.375980 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mp9l\" (UniqueName: \"kubernetes.io/projected/fe912d29-01ab-4b03-a703-613a525154d1-kube-api-access-2mp9l\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.376031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.375990 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd6zc\" (UniqueName: \"kubernetes.io/projected/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504-kube-api-access-gd6zc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:21.560756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.560725 5565 scope.go:122] "RemoveContainer" containerID="3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e" Oct 30 06:15:21.560962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.560866 5565 scope.go:122] "RemoveContainer" containerID="6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad" Oct 30 06:15:21.711028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.711003 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" Oct 30 06:15:21.711028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.711014 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f" event={"ID":"cbe50e49-14ed-4ad1-8ca6-213effbd0a45","Type":"ContainerDied","Data":"4176e54a961ea51576d974809fbf2b96626c30f1efa51f68e9eb6f4f49fd9d22"} Oct 30 06:15:21.711503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.711116 5565 scope.go:122] "RemoveContainer" containerID="9e2973db3588b8e0947a27682795a2c6c1cd0e08114824efa584eeef53f8170d" Oct 30 06:15:21.713563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.713500 5565 generic.go:356] "Generic (PLEG): container finished" podID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerID="a20aa8808c55a1fc0d662336e0b7ac7ce5227f763f7559bc95ccde09440a17ba" exitCode=60 Oct 30 06:15:21.713668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.713584 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" event={"ID":"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b","Type":"ContainerDied","Data":"a20aa8808c55a1fc0d662336e0b7ac7ce5227f763f7559bc95ccde09440a17ba"} Oct 30 06:15:21.714020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.714000 5565 scope.go:122] "RemoveContainer" containerID="a20aa8808c55a1fc0d662336e0b7ac7ce5227f763f7559bc95ccde09440a17ba" Oct 30 06:15:21.714338 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:21.714317 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv_odh-ai-gateway-infra(9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" Oct 30 06:15:21.715227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.715208 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" event={"ID":"62d68114-a3f3-48eb-b3ca-ff1bdbbd6504","Type":"ContainerDied","Data":"fcdcaddc3098c20d025f463b1e016496412361a7745692afcd5d7f413c70cd1f"} Oct 30 06:15:21.715303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.715228 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p" Oct 30 06:15:21.717296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.717276 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerStarted","Data":"0ae8cc8dba07c770b4ef17d92d4ae8f22cd6dbc0f3ad77ac2aafc4305d114bd2"} Oct 30 06:15:21.718827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.718812 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" Oct 30 06:15:21.718975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.718831 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg" event={"ID":"fe912d29-01ab-4b03-a703-613a525154d1","Type":"ContainerDied","Data":"d5117ee847d0a89de2b37966cb9cfc0de61bc683b8028ed1bb789a334c413743"} Oct 30 06:15:21.720986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.720966 5565 generic.go:356] "Generic (PLEG): container finished" podID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerID="b9377487081e7cdacabf206b59c5a76f7daa8a9a717a45b860068198ff1b41a4" exitCode=60 Oct 30 06:15:21.721089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.721062 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerDied","Data":"b9377487081e7cdacabf206b59c5a76f7daa8a9a717a45b860068198ff1b41a4"} Oct 30 06:15:21.721379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.721360 5565 scope.go:122] "RemoveContainer" containerID="b9377487081e7cdacabf206b59c5a76f7daa8a9a717a45b860068198ff1b41a4" Oct 30 06:15:21.721651 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:21.721628 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6_odh-ai-gateway-infra(af0c5f69-23fd-4032-a17d-e41ee3336e36)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" Oct 30 06:15:21.723712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.723692 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerStarted","Data":"12a48d1e1a4cf7dc47fb70483c477591c073be6926ef67e7709476dbd6cbb325"} Oct 30 06:15:21.726965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.725656 5565 generic.go:356] "Generic (PLEG): container finished" podID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerID="1a73a907514a83174ec5fc8e4253168f4bd75067067de0a0a897a08f80c22a87" exitCode=60 Oct 30 06:15:21.726965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.725725 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" event={"ID":"633df5e9-b8cf-480a-9b49-1a013f55e729","Type":"ContainerDied","Data":"1a73a907514a83174ec5fc8e4253168f4bd75067067de0a0a897a08f80c22a87"} Oct 30 06:15:21.726965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.725847 5565 scope.go:122] "RemoveContainer" containerID="14963ce2113066a12fecc6f87702a8fe1f6ade43f16950de420233332445a325" Oct 30 06:15:21.726965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.725987 5565 scope.go:122] "RemoveContainer" containerID="1a73a907514a83174ec5fc8e4253168f4bd75067067de0a0a897a08f80c22a87" Oct 30 06:15:21.726965 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:21.726210 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb_odh-ai-gateway-infra(633df5e9-b8cf-480a-9b49-1a013f55e729)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" Oct 30 06:15:21.742272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.742253 5565 scope.go:122] "RemoveContainer" containerID="5bffb1aa32c3f9e5ffb3b4bd503baf2860b254fc770ccac77c055b0ed7ce08b7" Oct 30 06:15:21.752775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.752757 5565 scope.go:122] "RemoveContainer" containerID="bed9d6a4e5a2b19b956d579c245b5fca2781f7fe7c156062c6c47e6c5caef4ca" Oct 30 06:15:21.874487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.874456 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb"] Oct 30 06:15:21.877310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.877281 5565 scope.go:122] "RemoveContainer" containerID="08cace4dff2346d5729a31a3cac60ca8202482357b1f8ffc43b97ec7460479c1" Oct 30 06:15:21.898716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.897912 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg"] Oct 30 06:15:21.906513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.906482 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-37bbcc-29889015-9sfzg"] Oct 30 06:15:21.910938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.910912 5565 scope.go:122] "RemoveContainer" containerID="41d2ef7695bffd00524780bf536f6ea5dd56be4fe902afc2d0ce91248782748c" Oct 30 06:15:21.922547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.922503 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6"] Oct 30 06:15:21.965415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.965349 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv"] Oct 30 06:15:21.988720 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:21.988691 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p"] Oct 30 06:15:22.000923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.000898 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-02ceed-29889015-jpd5p"] Oct 30 06:15:22.025019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.024997 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:22.027351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.027335 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:22.089981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.089958 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca\") pod \"ba63fa73-dbd3-466a-b288-9ed56a958e80\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " Oct 30 06:15:22.090123 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090110 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x\" (UniqueName: \"kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x\") pod \"ba63fa73-dbd3-466a-b288-9ed56a958e80\" (UID: \"ba63fa73-dbd3-466a-b288-9ed56a958e80\") " Oct 30 06:15:22.090163 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090149 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca\") pod \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " Oct 30 06:15:22.090268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090249 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca" pod "ba63fa73-dbd3-466a-b288-9ed56a958e80" (UID: "ba63fa73-dbd3-466a-b288-9ed56a958e80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:22.090321 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090269 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl\" (UniqueName: \"kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl\") pod \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\" (UID: \"3f2c5ea0-81b3-48c7-942e-7c064f1871bf\") " Oct 30 06:15:22.090615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090590 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca" pod "3f2c5ea0-81b3-48c7-942e-7c064f1871bf" (UID: "3f2c5ea0-81b3-48c7-942e-7c064f1871bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:22.090842 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090824 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.090917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.090843 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba63fa73-dbd3-466a-b288-9ed56a958e80-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.092224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.092200 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl" pod "3f2c5ea0-81b3-48c7-942e-7c064f1871bf" (UID: "3f2c5ea0-81b3-48c7-942e-7c064f1871bf"). InnerVolumeSpecName "kube-api-access-7tfjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:22.092307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.092237 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x" pod "ba63fa73-dbd3-466a-b288-9ed56a958e80" (UID: "ba63fa73-dbd3-466a-b288-9ed56a958e80"). InnerVolumeSpecName "kube-api-access-t5v2x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:22.168491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.168465 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f"] Oct 30 06:15:22.174488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.174457 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-d96a1f-29889015-4qw5f"] Oct 30 06:15:22.192509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.192485 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tfjl\" (UniqueName: \"kubernetes.io/projected/3f2c5ea0-81b3-48c7-942e-7c064f1871bf-kube-api-access-7tfjl\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.192509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.192508 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5v2x\" (UniqueName: \"kubernetes.io/projected/ba63fa73-dbd3-466a-b288-9ed56a958e80-kube-api-access-t5v2x\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:22.738428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.738396 5565 generic.go:356] "Generic (PLEG): container finished" podID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerID="729028f704b223bea343b93f2b43f2259da90dce8336ddc58d235da169c40fb4" exitCode=60 Oct 30 06:15:22.738916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.738468 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerDied","Data":"729028f704b223bea343b93f2b43f2259da90dce8336ddc58d235da169c40fb4"} Oct 30 06:15:22.738916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.738522 5565 scope.go:122] "RemoveContainer" containerID="3068831ad7bc1ad2b89905a535e373178c87160a119d603d62931a590199db2e" Oct 30 06:15:22.739021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.738969 5565 scope.go:122] "RemoveContainer" containerID="729028f704b223bea343b93f2b43f2259da90dce8336ddc58d235da169c40fb4" Oct 30 06:15:22.739744 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:22.739397 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl_odh-ai-gateway-infra(cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" Oct 30 06:15:22.740380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.740361 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" Oct 30 06:15:22.740482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.740378 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b" event={"ID":"3f2c5ea0-81b3-48c7-942e-7c064f1871bf","Type":"ContainerDied","Data":"27b3ba914ef6b4f9c251e7b6964ccf8854d15b89546ce7e965728a4cde5c5b2b"} Oct 30 06:15:22.752450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.752355 5565 generic.go:356] "Generic (PLEG): container finished" podID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerID="0ae8cc8dba07c770b4ef17d92d4ae8f22cd6dbc0f3ad77ac2aafc4305d114bd2" exitCode=60 Oct 30 06:15:22.752545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.752460 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerDied","Data":"0ae8cc8dba07c770b4ef17d92d4ae8f22cd6dbc0f3ad77ac2aafc4305d114bd2"} Oct 30 06:15:22.752943 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.752915 5565 scope.go:122] "RemoveContainer" containerID="0ae8cc8dba07c770b4ef17d92d4ae8f22cd6dbc0f3ad77ac2aafc4305d114bd2" Oct 30 06:15:22.753398 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:22.753301 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r_odh-ai-gateway-infra(7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" Oct 30 06:15:22.760498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.760474 5565 generic.go:356] "Generic (PLEG): container finished" podID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerID="12a48d1e1a4cf7dc47fb70483c477591c073be6926ef67e7709476dbd6cbb325" exitCode=60 Oct 30 06:15:22.760633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.760619 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerDied","Data":"12a48d1e1a4cf7dc47fb70483c477591c073be6926ef67e7709476dbd6cbb325"} Oct 30 06:15:22.761026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.761009 5565 scope.go:122] "RemoveContainer" containerID="12a48d1e1a4cf7dc47fb70483c477591c073be6926ef67e7709476dbd6cbb325" Oct 30 06:15:22.761352 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:22.761334 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg_odh-ai-gateway-infra(fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" Oct 30 06:15:22.762465 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.762444 5565 scope.go:122] "RemoveContainer" containerID="3a642a854de7ed59d95c6b241d28e64a8fdb07a805c2f91d61b8ea971b628af0" Oct 30 06:15:22.764190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.764167 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" Oct 30 06:15:22.764286 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.764186 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk" event={"ID":"ba63fa73-dbd3-466a-b288-9ed56a958e80","Type":"ContainerDied","Data":"05d0a0d00241a8d850d7717db810da885bf345422e6e7c09740abc03bcaa8b38"} Oct 30 06:15:22.766859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.766819 5565 generic.go:356] "Generic (PLEG): container finished" podID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerID="3918bc1b13b515e2007b3f50922d16bd594295235758ccb08d5cb7690aa660c4" exitCode=60 Oct 30 06:15:22.766979 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.766881 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" event={"ID":"196a2b1b-619c-416b-a839-e3b1fd15699f","Type":"ContainerDied","Data":"3918bc1b13b515e2007b3f50922d16bd594295235758ccb08d5cb7690aa660c4"} Oct 30 06:15:22.767386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.767364 5565 scope.go:122] "RemoveContainer" containerID="3918bc1b13b515e2007b3f50922d16bd594295235758ccb08d5cb7690aa660c4" Oct 30 06:15:22.767758 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:22.767735 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf_odh-ai-gateway-infra(196a2b1b-619c-416b-a839-e3b1fd15699f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" Oct 30 06:15:22.801700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.801665 5565 scope.go:122] "RemoveContainer" containerID="41d2298cd9a4da03cb5e7ad89bfb3986383df4005fa972abf9eb60e815694116" Oct 30 06:15:22.839755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.839477 5565 scope.go:122] "RemoveContainer" containerID="c80529296f4a639dcc7aeaaf9190655c982c643a5c50bddbf34312f807644414" Oct 30 06:15:22.864002 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.863965 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg"] Oct 30 06:15:22.881335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.881311 5565 scope.go:122] "RemoveContainer" containerID="36315b08ba8ca6353498511c85b1d8d5b46d181e5e46baaefde06c50b5811d26" Oct 30 06:15:22.895106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.895083 5565 scope.go:122] "RemoveContainer" containerID="6b208ade845313fea210ea687f26f237856e6752a54b7a13d70a47b3e1fe1cad" Oct 30 06:15:22.898464 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.898438 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b"] Oct 30 06:15:22.904225 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.904185 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-ebe61e-29889015-kp55b"] Oct 30 06:15:22.974862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.974831 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r"] Oct 30 06:15:22.976998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:22.976973 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:23.010176 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.010139 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z\" (UniqueName: \"kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z\") pod \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " Oct 30 06:15:23.010354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.010285 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca\") pod \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\" (UID: \"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b\") " Oct 30 06:15:23.010862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.010832 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca" pod "9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" (UID: "9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:23.011274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.011258 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.012739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.012710 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z" pod "9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" (UID: "9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b"). InnerVolumeSpecName "kube-api-access-5jk9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:23.076852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.076785 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:23.079712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.079689 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:23.111874 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.111844 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2\" (UniqueName: \"kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2\") pod \"af0c5f69-23fd-4032-a17d-e41ee3336e36\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " Oct 30 06:15:23.112040 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112014 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca\") pod \"633df5e9-b8cf-480a-9b49-1a013f55e729\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " Oct 30 06:15:23.112101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112052 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca\") pod \"af0c5f69-23fd-4032-a17d-e41ee3336e36\" (UID: \"af0c5f69-23fd-4032-a17d-e41ee3336e36\") " Oct 30 06:15:23.112174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112142 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5\" (UniqueName: \"kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5\") pod \"633df5e9-b8cf-480a-9b49-1a013f55e729\" (UID: \"633df5e9-b8cf-480a-9b49-1a013f55e729\") " Oct 30 06:15:23.112425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112401 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca" pod "633df5e9-b8cf-480a-9b49-1a013f55e729" (UID: "633df5e9-b8cf-480a-9b49-1a013f55e729"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:23.112542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112498 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca" pod "af0c5f69-23fd-4032-a17d-e41ee3336e36" (UID: "af0c5f69-23fd-4032-a17d-e41ee3336e36"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:23.112745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112727 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/633df5e9-b8cf-480a-9b49-1a013f55e729-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.112795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112747 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/af0c5f69-23fd-4032-a17d-e41ee3336e36-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.112795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.112758 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jk9z\" (UniqueName: \"kubernetes.io/projected/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b-kube-api-access-5jk9z\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.114186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.114165 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2" pod "af0c5f69-23fd-4032-a17d-e41ee3336e36" (UID: "af0c5f69-23fd-4032-a17d-e41ee3336e36"). InnerVolumeSpecName "kube-api-access-mzhd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:23.114373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.114352 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5" pod "633df5e9-b8cf-480a-9b49-1a013f55e729" (UID: "633df5e9-b8cf-480a-9b49-1a013f55e729"). InnerVolumeSpecName "kube-api-access-5k2f5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:23.149371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.149182 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk"] Oct 30 06:15:23.153925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.153902 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-c201cd-29889015-n76lk"] Oct 30 06:15:23.214362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.214328 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzhd2\" (UniqueName: \"kubernetes.io/projected/af0c5f69-23fd-4032-a17d-e41ee3336e36-kube-api-access-mzhd2\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.214362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.214355 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5k2f5\" (UniqueName: \"kubernetes.io/projected/633df5e9-b8cf-480a-9b49-1a013f55e729-kube-api-access-5k2f5\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:23.570424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.570391 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" path="/var/lib/kubelet/pods/3f2c5ea0-81b3-48c7-942e-7c064f1871bf/volumes" Oct 30 06:15:23.570956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.570937 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" path="/var/lib/kubelet/pods/62d68114-a3f3-48eb-b3ca-ff1bdbbd6504/volumes" Oct 30 06:15:23.571337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.571324 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" path="/var/lib/kubelet/pods/ba63fa73-dbd3-466a-b288-9ed56a958e80/volumes" Oct 30 06:15:23.571963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.571944 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" path="/var/lib/kubelet/pods/cbe50e49-14ed-4ad1-8ca6-213effbd0a45/volumes" Oct 30 06:15:23.572393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.572379 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe912d29-01ab-4b03-a703-613a525154d1" path="/var/lib/kubelet/pods/fe912d29-01ab-4b03-a703-613a525154d1/volumes" Oct 30 06:15:23.779168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.779129 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" event={"ID":"633df5e9-b8cf-480a-9b49-1a013f55e729","Type":"ContainerDied","Data":"23a34aa5abf610a919cbc0105a36f160fcd0c4b8d9b093f851a29ea8189a7022"} Oct 30 06:15:23.779717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.779148 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb" Oct 30 06:15:23.779717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.779224 5565 scope.go:122] "RemoveContainer" containerID="1a73a907514a83174ec5fc8e4253168f4bd75067067de0a0a897a08f80c22a87" Oct 30 06:15:23.789608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.789091 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" Oct 30 06:15:23.789608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.789075 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv" event={"ID":"9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b","Type":"ContainerDied","Data":"22863b802198e812922cbd4b5d66d93635964c94043e26686aa7fbf7a5c03551"} Oct 30 06:15:23.795004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.794981 5565 scope.go:122] "RemoveContainer" containerID="a20aa8808c55a1fc0d662336e0b7ac7ce5227f763f7559bc95ccde09440a17ba" Oct 30 06:15:23.797853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.796977 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl"] Oct 30 06:15:23.801978 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.801953 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" event={"ID":"af0c5f69-23fd-4032-a17d-e41ee3336e36","Type":"ContainerDied","Data":"84c42e62b53a3f8a44aa820abdfd97546efc19bb4610d07b81a5d8ee5613eb07"} Oct 30 06:15:23.802091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.802063 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6" Oct 30 06:15:23.822851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.822248 5565 scope.go:122] "RemoveContainer" containerID="b9377487081e7cdacabf206b59c5a76f7daa8a9a717a45b860068198ff1b41a4" Oct 30 06:15:23.952827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.952300 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb"] Oct 30 06:15:23.957693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.957664 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w6-720e9c-29889015-p7bpb"] Oct 30 06:15:23.998168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:23.998008 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:24.029885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.029857 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6"] Oct 30 06:15:24.044031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.044003 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-b4ca14-29889015-hjtz6"] Oct 30 06:15:24.052307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.052275 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf"] Oct 30 06:15:24.054350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.054323 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm\" (UniqueName: \"kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm\") pod \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " Oct 30 06:15:24.054502 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.054482 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca\") pod \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\" (UID: \"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1\") " Oct 30 06:15:24.056134 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.056091 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca" pod "7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" (UID: "7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:24.064410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.064381 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv"] Oct 30 06:15:24.065324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.064942 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm" pod "7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" (UID: "7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1"). InnerVolumeSpecName "kube-api-access-4hkkm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:24.076077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.076006 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:24.080391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.080368 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-36ae27-29889015-pxxrv"] Oct 30 06:15:24.110578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.110554 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:24.157264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.157230 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca" pod "fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" (UID: "fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:24.157425 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.157285 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca\") pod \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " Oct 30 06:15:24.157501 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.157448 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb\" (UniqueName: \"kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb\") pod \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " Oct 30 06:15:24.157658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.157559 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca\") pod \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\" (UID: \"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f\") " Oct 30 06:15:24.157877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.157851 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca" pod "cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" (UID: "cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:24.158015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.157929 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z\" (UniqueName: \"kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z\") pod \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\" (UID: \"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5\") " Oct 30 06:15:24.158656 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.158632 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.158781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.158661 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.158781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.158691 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hkkm\" (UniqueName: \"kubernetes.io/projected/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-kube-api-access-4hkkm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.158781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.158706 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.161849 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.161821 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb" pod "cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" (UID: "cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f"). InnerVolumeSpecName "kube-api-access-f5cwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:24.164207 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.164180 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z" pod "fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" (UID: "fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5"). InnerVolumeSpecName "kube-api-access-dgz7z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:24.245043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.245012 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:24.264538 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.264485 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8\" (UniqueName: \"kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8\") pod \"196a2b1b-619c-416b-a839-e3b1fd15699f\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " Oct 30 06:15:24.264726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.264684 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca\") pod \"196a2b1b-619c-416b-a839-e3b1fd15699f\" (UID: \"196a2b1b-619c-416b-a839-e3b1fd15699f\") " Oct 30 06:15:24.265304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.265278 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca" pod "196a2b1b-619c-416b-a839-e3b1fd15699f" (UID: "196a2b1b-619c-416b-a839-e3b1fd15699f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:24.266730 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.266692 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5cwb\" (UniqueName: \"kubernetes.io/projected/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f-kube-api-access-f5cwb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.266730 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.266724 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/196a2b1b-619c-416b-a839-e3b1fd15699f-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.266932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.266738 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgz7z\" (UniqueName: \"kubernetes.io/projected/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5-kube-api-access-dgz7z\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.268073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.268034 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8" pod "196a2b1b-619c-416b-a839-e3b1fd15699f" (UID: "196a2b1b-619c-416b-a839-e3b1fd15699f"). InnerVolumeSpecName "kube-api-access-65dn8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:24.368254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.368175 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65dn8\" (UniqueName: \"kubernetes.io/projected/196a2b1b-619c-416b-a839-e3b1fd15699f-kube-api-access-65dn8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:24.813834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.813803 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" Oct 30 06:15:24.814334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.813828 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf" event={"ID":"196a2b1b-619c-416b-a839-e3b1fd15699f","Type":"ContainerDied","Data":"98c5b404e0ed356cedec231f28cb3a99cc748afe421699b64a103d4eb05516ee"} Oct 30 06:15:24.814334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.813895 5565 scope.go:122] "RemoveContainer" containerID="3918bc1b13b515e2007b3f50922d16bd594295235758ccb08d5cb7690aa660c4" Oct 30 06:15:24.815358 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.815327 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" Oct 30 06:15:24.815492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.815381 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl" event={"ID":"cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f","Type":"ContainerDied","Data":"cfbf88529fe10b9c14133030c8ae5d8bfc43d32efef99025a3726123dab0eda7"} Oct 30 06:15:24.819214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.819194 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" Oct 30 06:15:24.819369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.819219 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r" event={"ID":"7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1","Type":"ContainerDied","Data":"58a6565927d0fa1a89f0960b19b64c4145aac239f3904be86189f03749ac8722"} Oct 30 06:15:24.828028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.825091 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" event={"ID":"fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5","Type":"ContainerDied","Data":"1421a980d9d80b1967adb2248ae7ffed9251b19e4709c1730afe1991c77d18e7"} Oct 30 06:15:24.828028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.825184 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg" Oct 30 06:15:24.828028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.827708 5565 scope.go:122] "RemoveContainer" containerID="729028f704b223bea343b93f2b43f2259da90dce8336ddc58d235da169c40fb4" Oct 30 06:15:24.839226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.839207 5565 scope.go:122] "RemoveContainer" containerID="0ae8cc8dba07c770b4ef17d92d4ae8f22cd6dbc0f3ad77ac2aafc4305d114bd2" Oct 30 06:15:24.849021 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.848990 5565 scope.go:122] "RemoveContainer" containerID="12a48d1e1a4cf7dc47fb70483c477591c073be6926ef67e7709476dbd6cbb325" Oct 30 06:15:24.872379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.872357 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl"] Oct 30 06:15:24.876042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.876022 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-452a59-29889015-hgfwl"] Oct 30 06:15:24.930991 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.930929 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf"] Oct 30 06:15:24.935978 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.935950 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-698ab2-29889015-9pgrf"] Oct 30 06:15:24.989393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.989363 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg"] Oct 30 06:15:24.995180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:24.995156 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-e7a506-29889015-td9vg"] Oct 30 06:15:25.034488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.034460 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r"] Oct 30 06:15:25.041479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.041451 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-64123f-29889015-kng5r"] Oct 30 06:15:25.569572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.569543 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" path="/var/lib/kubelet/pods/196a2b1b-619c-416b-a839-e3b1fd15699f/volumes" Oct 30 06:15:25.570063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.570049 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" path="/var/lib/kubelet/pods/633df5e9-b8cf-480a-9b49-1a013f55e729/volumes" Oct 30 06:15:25.570492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.570476 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" path="/var/lib/kubelet/pods/7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1/volumes" Oct 30 06:15:25.571245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.571231 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" path="/var/lib/kubelet/pods/9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b/volumes" Oct 30 06:15:25.571664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.571650 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" path="/var/lib/kubelet/pods/af0c5f69-23fd-4032-a17d-e41ee3336e36/volumes" Oct 30 06:15:25.572055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.572043 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" path="/var/lib/kubelet/pods/cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f/volumes" Oct 30 06:15:25.572731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:25.572716 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" path="/var/lib/kubelet/pods/fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5/volumes" Oct 30 06:15:28.946114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:28.946070 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst"] Oct 30 06:15:31.065005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:31.064969 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:15:31.065414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:31.064976 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:15:33.946610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:33.946544 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" podUID="a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" containerName="payload-pre-processing" containerID="cri-o://945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da" gracePeriod=25 Oct 30 06:15:34.295911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.295885 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:15:34.471866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.471835 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume\") pod \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " Oct 30 06:15:34.472016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.471932 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx\" (UniqueName: \"kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx\") pod \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\" (UID: \"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce\") " Oct 30 06:15:34.472268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.472243 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume" pod "a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" (UID: "a8d44c23-c3b7-478a-8fb1-022fd48ea3ce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:34.472554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.472521 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.474165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.474144 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx" pod "a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" (UID: "a8d44c23-c3b7-478a-8fb1-022fd48ea3ce"). InnerVolumeSpecName "kube-api-access-8cbkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:34.574400 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.574369 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cbkx\" (UniqueName: \"kubernetes.io/projected/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce-kube-api-access-8cbkx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.732868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.732839 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 06:15:34.749554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.749508 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 06:15:34.878945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.878862 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume\") pod \"950f39da-aad8-4ac4-93d2-98d47cf0789a\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " Oct 30 06:15:34.879101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.878953 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume\") pod \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " Oct 30 06:15:34.879101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.879029 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t\" (UniqueName: \"kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t\") pod \"950f39da-aad8-4ac4-93d2-98d47cf0789a\" (UID: \"950f39da-aad8-4ac4-93d2-98d47cf0789a\") " Oct 30 06:15:34.879101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.879071 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb\" (UniqueName: \"kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb\") pod \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\" (UID: \"29254e0c-3ea4-4b09-a9eb-4a199e28c85d\") " Oct 30 06:15:34.879369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.879345 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume" pod "29254e0c-3ea4-4b09-a9eb-4a199e28c85d" (UID: "29254e0c-3ea4-4b09-a9eb-4a199e28c85d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:34.879406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.879343 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume" pod "950f39da-aad8-4ac4-93d2-98d47cf0789a" (UID: "950f39da-aad8-4ac4-93d2-98d47cf0789a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:15:34.879793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.879774 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/950f39da-aad8-4ac4-93d2-98d47cf0789a-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.879863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.879795 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.881221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.881198 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb" pod "29254e0c-3ea4-4b09-a9eb-4a199e28c85d" (UID: "29254e0c-3ea4-4b09-a9eb-4a199e28c85d"). InnerVolumeSpecName "kube-api-access-htzxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:34.881283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.881222 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t" pod "950f39da-aad8-4ac4-93d2-98d47cf0789a" (UID: "950f39da-aad8-4ac4-93d2-98d47cf0789a"). InnerVolumeSpecName "kube-api-access-c4n4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:15:34.918340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.918309 5565 generic.go:356] "Generic (PLEG): container finished" podID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerID="4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315" exitCode=0 Oct 30 06:15:34.918480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.918357 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" Oct 30 06:15:34.918480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.918404 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" event={"ID":"950f39da-aad8-4ac4-93d2-98d47cf0789a","Type":"ContainerDied","Data":"4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315"} Oct 30 06:15:34.918480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.918458 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl" event={"ID":"950f39da-aad8-4ac4-93d2-98d47cf0789a","Type":"ContainerDied","Data":"bd2dfe26cde2ee6f55541a34cfad336c990f238e2487e85d0a983e94d47aec1f"} Oct 30 06:15:34.918683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.918488 5565 scope.go:122] "RemoveContainer" containerID="4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315" Oct 30 06:15:34.919983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.919959 5565 generic.go:356] "Generic (PLEG): container finished" podID="a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" containerID="945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da" exitCode=0 Oct 30 06:15:34.920099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.919993 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" Oct 30 06:15:34.920099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.920031 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" event={"ID":"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce","Type":"ContainerDied","Data":"945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da"} Oct 30 06:15:34.920099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.920085 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst" event={"ID":"a8d44c23-c3b7-478a-8fb1-022fd48ea3ce","Type":"ContainerDied","Data":"2698bedf7a43e65442248fcf37b64d5302db4b0c25a348caf6ccd5a64d531c5e"} Oct 30 06:15:34.921791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.921764 5565 generic.go:356] "Generic (PLEG): container finished" podID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerID="9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198" exitCode=0 Oct 30 06:15:34.921905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.921813 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" Oct 30 06:15:34.921905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.921830 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" event={"ID":"29254e0c-3ea4-4b09-a9eb-4a199e28c85d","Type":"ContainerDied","Data":"9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198"} Oct 30 06:15:34.921905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.921869 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47" event={"ID":"29254e0c-3ea4-4b09-a9eb-4a199e28c85d","Type":"ContainerDied","Data":"d95e8cd8d74b817d52555b3df7a8b8b3a8a54804e9081ad137e368962c02a717"} Oct 30 06:15:34.932508 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.932485 5565 scope.go:122] "RemoveContainer" containerID="4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315" Oct 30 06:15:34.932860 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:34.932837 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315\": container with ID starting with 4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315 not found: ID does not exist" containerID="4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315" Oct 30 06:15:34.933060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.932865 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315"} err="failed to get container status \"4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315\": rpc error: code = NotFound desc = could not find container \"4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315\": container with ID starting with 4142f4d2f2c9fd695151f179926abffb1d1daa1aa1b2a7008442de58f10e1315 not found: ID does not exist" Oct 30 06:15:34.933060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.932883 5565 scope.go:122] "RemoveContainer" containerID="945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da" Oct 30 06:15:34.943451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.943431 5565 scope.go:122] "RemoveContainer" containerID="945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da" Oct 30 06:15:34.943755 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:34.943736 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da\": container with ID starting with 945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da not found: ID does not exist" containerID="945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da" Oct 30 06:15:34.943843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.943764 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da"} err="failed to get container status \"945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da\": rpc error: code = NotFound desc = could not find container \"945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da\": container with ID starting with 945396c2670bfde3919c36e116816f195bb7e267712882c61dbe5ce8a54197da not found: ID does not exist" Oct 30 06:15:34.943843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.943782 5565 scope.go:122] "RemoveContainer" containerID="9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198" Oct 30 06:15:34.954916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.954735 5565 scope.go:122] "RemoveContainer" containerID="9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198" Oct 30 06:15:34.955189 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:34.955059 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198\": container with ID starting with 9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198 not found: ID does not exist" containerID="9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198" Oct 30 06:15:34.955189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.955086 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198"} err="failed to get container status \"9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198\": rpc error: code = NotFound desc = could not find container \"9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198\": container with ID starting with 9ccbe4e47d0db238ba0c23ddf23da3c7d94262feaf190e06521098a3fa68c198 not found: ID does not exist" Oct 30 06:15:34.978503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.978476 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47"] Oct 30 06:15:34.981612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.981589 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4n4t\" (UniqueName: \"kubernetes.io/projected/950f39da-aad8-4ac4-93d2-98d47cf0789a-kube-api-access-c4n4t\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.981612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.981614 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htzxb\" (UniqueName: \"kubernetes.io/projected/29254e0c-3ea4-4b09-a9eb-4a199e28c85d-kube-api-access-htzxb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:15:34.985002 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:34.984980 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-b4ca14-6ff86845b-6fk47"] Oct 30 06:15:35.022979 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.022948 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst"] Oct 30 06:15:35.026556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.026518 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-84c5969895-5rrst"] Oct 30 06:15:35.064242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.064207 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl"] Oct 30 06:15:35.067981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.067955 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-b4ca14-65c7bb64dc-28sjl"] Oct 30 06:15:35.569897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.569859 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" path="/var/lib/kubelet/pods/29254e0c-3ea4-4b09-a9eb-4a199e28c85d/volumes" Oct 30 06:15:35.570458 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.570436 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" path="/var/lib/kubelet/pods/950f39da-aad8-4ac4-93d2-98d47cf0789a/volumes" Oct 30 06:15:35.570945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:35.570925 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" path="/var/lib/kubelet/pods/a8d44c23-c3b7-478a-8fb1-022fd48ea3ce/volumes" Oct 30 06:15:38.494173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:38.494141 5565 scope.go:122] "RemoveContainer" containerID="88477d2c46eb2454b0c1296b836cd0022294ab0d783c5b6da7f8702ae3b847c7" Oct 30 06:15:38.504659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:38.504595 5565 scope.go:122] "RemoveContainer" containerID="f2583927bfa98bd4785c9027551ba2ad7917513d9c2f4b2d3fa346897590db7b" Oct 30 06:15:41.031745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:41.031714 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5"] Oct 30 06:15:41.045625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:41.045593 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-87691408-4zhx5"] Oct 30 06:15:41.569022 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:41.568986 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bfdf5604-7b91-4885-9224-722b777e9f1a" path="/var/lib/kubelet/pods/bfdf5604-7b91-4885-9224-722b777e9f1a/volumes" Oct 30 06:15:44.853559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.853508 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm"] Oct 30 06:15:44.854338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854319 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854339 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854351 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854356 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854364 5565 cpu_manager.go:414] "containerMap: removing container" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854369 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854378 5565 cpu_manager.go:414] "containerMap: removing container" podUID="81e69693-69c0-4d07-867e-8d385f41c342" containerName="payload-pre-processing" Oct 30 06:15:44.854383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854383 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81e69693-69c0-4d07-867e-8d385f41c342" containerName="payload-pre-processing" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854391 5565 cpu_manager.go:414] "containerMap: removing container" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854396 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854405 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854409 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854419 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854423 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854434 5565 cpu_manager.go:414] "containerMap: removing container" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854438 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854445 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854450 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854457 5565 cpu_manager.go:414] "containerMap: removing container" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854461 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854467 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854471 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854482 5565 cpu_manager.go:414] "containerMap: removing container" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerName="payload-processing" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854486 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerName="payload-processing" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854492 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854497 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854505 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854509 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854516 5565 cpu_manager.go:414] "containerMap: removing container" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854520 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854544 5565 cpu_manager.go:414] "containerMap: removing container" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854549 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854559 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854566 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854577 5565 cpu_manager.go:414] "containerMap: removing container" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerName="payload-pre-processing" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854584 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerName="payload-pre-processing" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854593 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854600 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854616 5565 cpu_manager.go:414] "containerMap: removing container" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854621 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854629 5565 cpu_manager.go:414] "containerMap: removing container" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854634 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854642 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854646 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854654 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854659 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854665 5565 cpu_manager.go:414] "containerMap: removing container" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854669 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854678 5565 cpu_manager.go:414] "containerMap: removing container" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854682 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854691 5565 cpu_manager.go:414] "containerMap: removing container" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854696 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854703 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854708 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854716 5565 cpu_manager.go:414] "containerMap: removing container" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854720 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854726 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:15:44.854712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854731 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854739 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerName="payload-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854744 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerName="payload-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854756 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854760 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854769 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" containerName="payload-pre-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854773 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" containerName="payload-pre-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854779 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854783 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854789 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854792 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854911 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854919 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854926 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854932 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854938 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854943 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854948 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854953 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854958 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854963 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854970 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fda9767-1e03-4e98-800f-e9ed34dedf47" containerName="payload-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854976 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854981 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854986 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8d44c23-c3b7-478a-8fb1-022fd48ea3ce" containerName="payload-pre-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854992 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.854998 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855006 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855012 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855017 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855022 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="81e69693-69c0-4d07-867e-8d385f41c342" containerName="payload-pre-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855028 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855033 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855040 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855045 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855051 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="29254e0c-3ea4-4b09-a9eb-4a199e28c85d" containerName="payload-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855057 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855064 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855070 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="950f39da-aad8-4ac4-93d2-98d47cf0789a" containerName="payload-pre-processing" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855077 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855082 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:15:44.856457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.855088 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="633df5e9-b8cf-480a-9b49-1a013f55e729" containerName="cleanup" Oct 30 06:15:44.861758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.861738 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:44.879768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.879740 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm"] Oct 30 06:15:44.889696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.889650 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb9345-dockercfg-hg6g2\"" Oct 30 06:15:44.889696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:44.889653 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb9345\"" Oct 30 06:15:45.060552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.024597 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.060552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.024952 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6gswv\" (UniqueName: \"kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv\") pod \"payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.132602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.130898 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6gswv\" (UniqueName: \"kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv\") pod \"payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.132602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.131113 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.132602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.132196 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.177436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.177406 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6gswv\" (UniqueName: \"kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv\") pod \"payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.477793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.477703 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:45.791993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:45.791958 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm"] Oct 30 06:15:45.792951 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:15:45.792922 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod146e5605_cb4f_410e_ae48_b2e8fbdedd4a.slice/crio-225e9d48d668e2580406269844f248ad2246a321f34ed4e2a7b66bf016dcd9b3 WatchSource:0}: Error finding container 225e9d48d668e2580406269844f248ad2246a321f34ed4e2a7b66bf016dcd9b3: Status 404 returned error can't find the container with id 225e9d48d668e2580406269844f248ad2246a321f34ed4e2a7b66bf016dcd9b3 Oct 30 06:15:46.096229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:46.096141 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" event={"ID":"146e5605-cb4f-410e-ae48-b2e8fbdedd4a","Type":"ContainerStarted","Data":"2d2442c92aaf02336e52859bb397a444591748ba2781da5a7b9c4220bbbb6b87"} Oct 30 06:15:46.096229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:46.096194 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" event={"ID":"146e5605-cb4f-410e-ae48-b2e8fbdedd4a","Type":"ContainerStarted","Data":"225e9d48d668e2580406269844f248ad2246a321f34ed4e2a7b66bf016dcd9b3"} Oct 30 06:15:46.096229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:46.096224 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:15:46.144033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:46.143982 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" podStartSLOduration=2.143967361 podStartE2EDuration="2.143967361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:15:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:15:46.141560848 +0000 UTC m=+2652.717247903" watchObservedRunningTime="2026-10-30 06:15:46.143967361 +0000 UTC m=+2652.719654472" Oct 30 06:15:47.534116 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:15:47.534073 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:15:50.847206 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:50.847171 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:15:52.053065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:52.053017 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7"] Oct 30 06:15:52.062547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:52.059198 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-663d6682-l5nn7"] Oct 30 06:15:53.569862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:53.569826 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4" path="/var/lib/kubelet/pods/c94cc0f9-ead4-4be9-9e19-9ed3eb53d6d4/volumes" Oct 30 06:15:57.109349 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:15:57.109320 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:16:02.033862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:02.033813 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl"] Oct 30 06:16:02.047541 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:02.047498 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8601f6-h97hl"] Oct 30 06:16:03.571009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:03.570972 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="060da714-74a7-48a0-8298-690d068f558a" path="/var/lib/kubelet/pods/060da714-74a7-48a0-8298-690d068f558a/volumes" Oct 30 06:16:03.893507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:03.893398 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8"] Oct 30 06:16:03.895255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:03.895215 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" containerID="cri-o://af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456" gracePeriod=30 Oct 30 06:16:04.542494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.542450 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd"] Oct 30 06:16:04.543723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543697 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:16:04.543723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543722 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543745 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543751 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543762 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543768 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe912d29-01ab-4b03-a703-613a525154d1" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543783 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543791 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543800 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543804 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543826 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543831 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543844 5565 cpu_manager.go:414] "containerMap: removing container" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:16:04.543916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543849 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543981 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe50e49-14ed-4ad1-8ca6-213effbd0a45" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.543993 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="62d68114-a3f3-48eb-b3ca-ff1bdbbd6504" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544000 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="196a2b1b-619c-416b-a839-e3b1fd15699f" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544009 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f2c5ea0-81b3-48c7-942e-7c064f1871bf" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544017 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cfaa8ae0-f2da-4da9-9a03-5986b46d2b0f" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544023 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd42b394-fdcd-4b5d-8061-eeb4b7bee7b5" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544031 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba63fa73-dbd3-466a-b288-9ed56a958e80" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544036 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="af0c5f69-23fd-4032-a17d-e41ee3336e36" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544045 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c728aa6-b4b2-4a17-aeee-84d84f2f9a1b" containerName="cleanup" Oct 30 06:16:04.544294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.544052 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7c1c7f37-ea41-4ce8-ab02-7d68b23f57c1" containerName="cleanup" Oct 30 06:16:04.548148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.548121 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.571709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.571642 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd"] Oct 30 06:16:04.638927 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.638890 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:04.639160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.638943 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:16:04.645179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.645140 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.645572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.645521 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.645896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.645865 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdxkk\" (UniqueName: \"kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.747651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.747614 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.747885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.747708 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdxkk\" (UniqueName: \"kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.747885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.747798 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.748576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.748511 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.750928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.750891 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.773417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.773374 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdxkk\" (UniqueName: \"kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk\") pod \"maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:04.877199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:04.877097 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:05.132585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.132469 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56"] Oct 30 06:16:05.137736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.137707 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.160473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.160427 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56"] Oct 30 06:16:05.228420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.228384 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd"] Oct 30 06:16:05.229650 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:05.229619 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7240fea0_9d52_4f59_9c3a_5ef205ebf95c.slice/crio-ada6c7d52343fdc777a085a8479b7f0e2c20aea5b43c3ce94927460af9a209ed WatchSource:0}: Error finding container ada6c7d52343fdc777a085a8479b7f0e2c20aea5b43c3ce94927460af9a209ed: Status 404 returned error can't find the container with id ada6c7d52343fdc777a085a8479b7f0e2c20aea5b43c3ce94927460af9a209ed Oct 30 06:16:05.266036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.265994 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.266211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.266053 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qscdc\" (UniqueName: \"kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.266277 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.266234 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.297665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.297626 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" event={"ID":"7240fea0-9d52-4f59-9c3a-5ef205ebf95c","Type":"ContainerStarted","Data":"ada6c7d52343fdc777a085a8479b7f0e2c20aea5b43c3ce94927460af9a209ed"} Oct 30 06:16:05.367644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.367598 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.367825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.367663 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qscdc\" (UniqueName: \"kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.367825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.367797 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.368372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.368348 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.454055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.453957 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.454234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.454211 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qscdc\" (UniqueName: \"kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc\") pod \"maas-api-revoke-sub-cf777683-5jw56\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.456808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.456774 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:05.773231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:05.773201 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56"] Oct 30 06:16:05.773716 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:05.773329 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219e908c_5a9a_4b73_84db_359cf49e5bae.slice/crio-1fc871dd597489daabeafa92cd5267addfcde9b797413038f52d459f26c8e012 WatchSource:0}: Error finding container 1fc871dd597489daabeafa92cd5267addfcde9b797413038f52d459f26c8e012: Status 404 returned error can't find the container with id 1fc871dd597489daabeafa92cd5267addfcde9b797413038f52d459f26c8e012 Oct 30 06:16:06.309173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:06.309131 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" event={"ID":"7240fea0-9d52-4f59-9c3a-5ef205ebf95c","Type":"ContainerStarted","Data":"7dd899d8ba3360f8b79b52f8d0f51c8e28a21e9cf538647a74031a7aaca108b8"} Oct 30 06:16:06.310980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:06.310952 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" event={"ID":"219e908c-5a9a-4b73-84db-359cf49e5bae","Type":"ContainerStarted","Data":"6c8dde065c663a55ac701415d984b38358730e96b7e3353a65c4c7778176d51d"} Oct 30 06:16:06.311083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:06.311004 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" event={"ID":"219e908c-5a9a-4b73-84db-359cf49e5bae","Type":"ContainerStarted","Data":"1fc871dd597489daabeafa92cd5267addfcde9b797413038f52d459f26c8e012"} Oct 30 06:16:06.355320 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:06.355261 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" podStartSLOduration=2.3552407300000002 podStartE2EDuration="2.35524073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:06.354020631 +0000 UTC m=+2672.929707686" watchObservedRunningTime="2026-10-30 06:16:06.35524073 +0000 UTC m=+2672.930927789" Oct 30 06:16:06.401488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:06.401397 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" podStartSLOduration=1.4013730340000001 podStartE2EDuration="1.401373034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:06.397783538 +0000 UTC m=+2672.973470591" watchObservedRunningTime="2026-10-30 06:16:06.401373034 +0000 UTC m=+2672.977060088" Oct 30 06:16:07.320120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:07.320083 5565 generic.go:356] "Generic (PLEG): container finished" podID="7240fea0-9d52-4f59-9c3a-5ef205ebf95c" containerID="7dd899d8ba3360f8b79b52f8d0f51c8e28a21e9cf538647a74031a7aaca108b8" exitCode=0 Oct 30 06:16:07.320621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:07.320176 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" event={"ID":"7240fea0-9d52-4f59-9c3a-5ef205ebf95c","Type":"ContainerDied","Data":"7dd899d8ba3360f8b79b52f8d0f51c8e28a21e9cf538647a74031a7aaca108b8"} Oct 30 06:16:07.322055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:07.322028 5565 generic.go:356] "Generic (PLEG): container finished" podID="219e908c-5a9a-4b73-84db-359cf49e5bae" containerID="6c8dde065c663a55ac701415d984b38358730e96b7e3353a65c4c7778176d51d" exitCode=0 Oct 30 06:16:07.322178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:07.322109 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" event={"ID":"219e908c-5a9a-4b73-84db-359cf49e5bae","Type":"ContainerDied","Data":"6c8dde065c663a55ac701415d984b38358730e96b7e3353a65c4c7778176d51d"} Oct 30 06:16:08.524246 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.524216 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:08.575341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.575302 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:08.620741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.620619 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc\" (UniqueName: \"kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc\") pod \"219e908c-5a9a-4b73-84db-359cf49e5bae\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " Oct 30 06:16:08.621115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.621087 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca\") pod \"219e908c-5a9a-4b73-84db-359cf49e5bae\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " Oct 30 06:16:08.621468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.621443 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca\") pod \"219e908c-5a9a-4b73-84db-359cf49e5bae\" (UID: \"219e908c-5a9a-4b73-84db-359cf49e5bae\") " Oct 30 06:16:08.621738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.621517 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca" pod "219e908c-5a9a-4b73-84db-359cf49e5bae" (UID: "219e908c-5a9a-4b73-84db-359cf49e5bae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:08.622439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.622404 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.623678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.623646 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc" pod "219e908c-5a9a-4b73-84db-359cf49e5bae" (UID: "219e908c-5a9a-4b73-84db-359cf49e5bae"). InnerVolumeSpecName "kube-api-access-qscdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:08.624022 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.623995 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca" pod "219e908c-5a9a-4b73-84db-359cf49e5bae" (UID: "219e908c-5a9a-4b73-84db-359cf49e5bae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:08.723905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.723863 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk\" (UniqueName: \"kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk\") pod \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " Oct 30 06:16:08.724116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.723950 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca\") pod \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " Oct 30 06:16:08.724116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.724093 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca\") pod \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\" (UID: \"7240fea0-9d52-4f59-9c3a-5ef205ebf95c\") " Oct 30 06:16:08.724496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.724468 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca" pod "7240fea0-9d52-4f59-9c3a-5ef205ebf95c" (UID: "7240fea0-9d52-4f59-9c3a-5ef205ebf95c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:08.724735 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.724715 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/219e908c-5a9a-4b73-84db-359cf49e5bae-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.724800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.724740 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.724800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.724756 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qscdc\" (UniqueName: \"kubernetes.io/projected/219e908c-5a9a-4b73-84db-359cf49e5bae-kube-api-access-qscdc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.726215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.726192 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca" pod "7240fea0-9d52-4f59-9c3a-5ef205ebf95c" (UID: "7240fea0-9d52-4f59-9c3a-5ef205ebf95c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:08.726281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.726225 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk" pod "7240fea0-9d52-4f59-9c3a-5ef205ebf95c" (UID: "7240fea0-9d52-4f59-9c3a-5ef205ebf95c"). InnerVolumeSpecName "kube-api-access-bdxkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:08.826017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.825969 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:08.826017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:08.826008 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdxkk\" (UniqueName: \"kubernetes.io/projected/7240fea0-9d52-4f59-9c3a-5ef205ebf95c-kube-api-access-bdxkk\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.164967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.164935 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 06:16:09.334420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334383 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334491 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334545 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334593 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334672 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334783 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.334981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334877 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334892 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.334981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334936 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.334981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.334945 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.335258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.335044 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:09.335258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.335105 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.335258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.335242 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.335417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.335180 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo\") pod \"2d65898f-a620-49fa-8930-eba9e5e61287\" (UID: \"2d65898f-a620-49fa-8930-eba9e5e61287\") " Oct 30 06:16:09.336282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.336254 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d65898f-a620-49fa-8930-eba9e5e61287-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.336432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.336284 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.336432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.336299 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.336432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.336314 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.336432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.336334 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.337897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.337856 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:09.338618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.338591 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:09.339088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.339055 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:16:09.340271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.340247 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf" pod "2d65898f-a620-49fa-8930-eba9e5e61287" (UID: "2d65898f-a620-49fa-8930-eba9e5e61287"). InnerVolumeSpecName "kube-api-access-mqmgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:09.343077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.343047 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" event={"ID":"7240fea0-9d52-4f59-9c3a-5ef205ebf95c","Type":"ContainerDied","Data":"ada6c7d52343fdc777a085a8479b7f0e2c20aea5b43c3ce94927460af9a209ed"} Oct 30 06:16:09.343186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.343111 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ada6c7d52343fdc777a085a8479b7f0e2c20aea5b43c3ce94927460af9a209ed" Oct 30 06:16:09.343186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.343060 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd" Oct 30 06:16:09.350448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.350422 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg"] Oct 30 06:16:09.352057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.352040 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" Oct 30 06:16:09.352879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.352077 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56" event={"ID":"219e908c-5a9a-4b73-84db-359cf49e5bae","Type":"ContainerDied","Data":"1fc871dd597489daabeafa92cd5267addfcde9b797413038f52d459f26c8e012"} Oct 30 06:16:09.352879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.352732 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fc871dd597489daabeafa92cd5267addfcde9b797413038f52d459f26c8e012" Oct 30 06:16:09.356056 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.356028 5565 generic.go:356] "Generic (PLEG): container finished" podID="2d65898f-a620-49fa-8930-eba9e5e61287" containerID="af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456" exitCode=0 Oct 30 06:16:09.356292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.356107 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" Oct 30 06:16:09.356292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.356145 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" event={"ID":"2d65898f-a620-49fa-8930-eba9e5e61287","Type":"ContainerDied","Data":"af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456"} Oct 30 06:16:09.356292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.356186 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8" event={"ID":"2d65898f-a620-49fa-8930-eba9e5e61287","Type":"ContainerDied","Data":"1c2f0e689e332a5e572e63049dee8655a77972b916fa15ef5e794fba5123486b"} Oct 30 06:16:09.356292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.356217 5565 scope.go:122] "RemoveContainer" containerID="af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456" Oct 30 06:16:09.358904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.358880 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd"] Oct 30 06:16:09.373777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.373749 5565 scope.go:122] "RemoveContainer" containerID="35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927" Oct 30 06:16:09.384592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.384568 5565 scope.go:122] "RemoveContainer" containerID="af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456" Oct 30 06:16:09.384889 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:09.384867 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456\": container with ID starting with af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456 not found: ID does not exist" containerID="af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456" Oct 30 06:16:09.384960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.384899 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456"} err="failed to get container status \"af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456\": rpc error: code = NotFound desc = could not find container \"af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456\": container with ID starting with af4b73e538ad370248859c290e0fe03fbe0ce673754b290189f119a60667e456 not found: ID does not exist" Oct 30 06:16:09.384960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.384916 5565 scope.go:122] "RemoveContainer" containerID="35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927" Oct 30 06:16:09.385161 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:09.385145 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927\": container with ID starting with 35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927 not found: ID does not exist" containerID="35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927" Oct 30 06:16:09.385199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.385168 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927"} err="failed to get container status \"35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927\": rpc error: code = NotFound desc = could not find container \"35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927\": container with ID starting with 35b395050ca46b6fb6f841db6f43de3e18b444f15c547ab78fa49caada72d927 not found: ID does not exist" Oct 30 06:16:09.438326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.438294 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.438326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.438322 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqmgf\" (UniqueName: \"kubernetes.io/projected/2d65898f-a620-49fa-8930-eba9e5e61287-kube-api-access-mqmgf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.438326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.438333 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d65898f-a620-49fa-8930-eba9e5e61287-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.438626 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.438344 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d65898f-a620-49fa-8930-eba9e5e61287-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:09.445042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.445011 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8"] Oct 30 06:16:09.453669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.453639 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-b4ca14-openshift-default-8564666b47-g4dc8"] Oct 30 06:16:09.570197 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:09.570161 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" path="/var/lib/kubelet/pods/2d65898f-a620-49fa-8930-eba9e5e61287/volumes" Oct 30 06:16:11.544008 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.543966 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b"] Oct 30 06:16:11.545671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.545642 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" Oct 30 06:16:11.545671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.545669 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" Oct 30 06:16:11.545883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.545760 5565 cpu_manager.go:414] "containerMap: removing container" podUID="219e908c-5a9a-4b73-84db-359cf49e5bae" containerName="revoke-keys" Oct 30 06:16:11.545883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.545770 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="219e908c-5a9a-4b73-84db-359cf49e5bae" containerName="revoke-keys" Oct 30 06:16:11.545883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.545823 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7240fea0-9d52-4f59-9c3a-5ef205ebf95c" containerName="revoke-keys" Oct 30 06:16:11.545883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.545851 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7240fea0-9d52-4f59-9c3a-5ef205ebf95c" containerName="revoke-keys" Oct 30 06:16:11.546231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.546214 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="219e908c-5a9a-4b73-84db-359cf49e5bae" containerName="revoke-keys" Oct 30 06:16:11.546294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.546249 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" Oct 30 06:16:11.546294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.546266 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7240fea0-9d52-4f59-9c3a-5ef205ebf95c" containerName="revoke-keys" Oct 30 06:16:11.546294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.546290 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" Oct 30 06:16:11.553260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.553237 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.574020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.573993 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:11.574690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.574664 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:16:11.576523 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.576499 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b"] Oct 30 06:16:11.667995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.667967 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.668137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.668038 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.668280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.668256 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhq48\" (UniqueName: \"kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.769088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.769052 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhq48\" (UniqueName: \"kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.769240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.769121 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.769240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.769223 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.769788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.769766 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.771610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.771589 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.819828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.819748 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhq48\" (UniqueName: \"kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48\") pod \"maas-api-revoke-sub-c59c6eb0-cbb9b\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:11.870943 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:11.870905 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:12.180112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:12.180081 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b"] Oct 30 06:16:12.180112 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:12.180096 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bfa5f9_77e2_478e_bfa1_4d0163b757ab.slice/crio-e4d4a8bee9c7dcf8396e5948f71f6137a3411f0734980033354212e4c337252a WatchSource:0}: Error finding container e4d4a8bee9c7dcf8396e5948f71f6137a3411f0734980033354212e4c337252a: Status 404 returned error can't find the container with id e4d4a8bee9c7dcf8396e5948f71f6137a3411f0734980033354212e4c337252a Oct 30 06:16:12.381672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:12.381641 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" event={"ID":"07bfa5f9-77e2-478e-bfa1-4d0163b757ab","Type":"ContainerStarted","Data":"e4d4a8bee9c7dcf8396e5948f71f6137a3411f0734980033354212e4c337252a"} Oct 30 06:16:13.031334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:13.031296 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9"] Oct 30 06:16:13.044995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:13.044962 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6a98ec1-8qkf9"] Oct 30 06:16:13.390958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:13.390861 5565 generic.go:356] "Generic (PLEG): container finished" podID="07bfa5f9-77e2-478e-bfa1-4d0163b757ab" containerID="8abb0771aea53fd4bc4d7fafe23c3fca24bd60be21cc99889e2016f23fa1d050" exitCode=0 Oct 30 06:16:13.390958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:13.390944 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" event={"ID":"07bfa5f9-77e2-478e-bfa1-4d0163b757ab","Type":"ContainerDied","Data":"8abb0771aea53fd4bc4d7fafe23c3fca24bd60be21cc99889e2016f23fa1d050"} Oct 30 06:16:13.572136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:13.572102 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04b1717a-35ca-47a6-99fc-ea41b86cb336" path="/var/lib/kubelet/pods/04b1717a-35ca-47a6-99fc-ea41b86cb336/volumes" Oct 30 06:16:14.590067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.590030 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:14.622612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.622522 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48\" (UniqueName: \"kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48\") pod \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " Oct 30 06:16:14.622833 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.622691 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca\") pod \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " Oct 30 06:16:14.623083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.623060 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca\") pod \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\" (UID: \"07bfa5f9-77e2-478e-bfa1-4d0163b757ab\") " Oct 30 06:16:14.623144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.623085 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca" pod "07bfa5f9-77e2-478e-bfa1-4d0163b757ab" (UID: "07bfa5f9-77e2-478e-bfa1-4d0163b757ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:14.624063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.624035 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:14.626941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.626903 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca" pod "07bfa5f9-77e2-478e-bfa1-4d0163b757ab" (UID: "07bfa5f9-77e2-478e-bfa1-4d0163b757ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:14.627248 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.627223 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48" pod "07bfa5f9-77e2-478e-bfa1-4d0163b757ab" (UID: "07bfa5f9-77e2-478e-bfa1-4d0163b757ab"). InnerVolumeSpecName "kube-api-access-rhq48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:14.726389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.726281 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:14.726389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.726325 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhq48\" (UniqueName: \"kubernetes.io/projected/07bfa5f9-77e2-478e-bfa1-4d0163b757ab-kube-api-access-rhq48\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:14.938379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.938302 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5"] Oct 30 06:16:14.990961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:14.990849 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b"] Oct 30 06:16:15.144558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.144493 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24"] Oct 30 06:16:15.145370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.145351 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" Oct 30 06:16:15.145417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.145371 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d65898f-a620-49fa-8930-eba9e5e61287" containerName="istio-proxy" Oct 30 06:16:15.145417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.145404 5565 cpu_manager.go:414] "containerMap: removing container" podUID="07bfa5f9-77e2-478e-bfa1-4d0163b757ab" containerName="revoke-keys" Oct 30 06:16:15.145417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.145411 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bfa5f9-77e2-478e-bfa1-4d0163b757ab" containerName="revoke-keys" Oct 30 06:16:15.145522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.145512 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bfa5f9-77e2-478e-bfa1-4d0163b757ab" containerName="revoke-keys" Oct 30 06:16:15.149749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.149722 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.167789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.167740 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-370dfa-dockercfg-vf9ml\"" Oct 30 06:16:15.169809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.169778 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-370dfa\"" Oct 30 06:16:15.171649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.171615 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24"] Oct 30 06:16:15.340638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.324344 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bgbq\" (UniqueName: \"kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.340638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.332091 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.443554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.443085 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bgbq\" (UniqueName: \"kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.443554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.443345 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.446784 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.446444 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.467568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.467406 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm"] Oct 30 06:16:15.573651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.563063 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" Oct 30 06:16:15.573651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.571999 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b" event={"ID":"07bfa5f9-77e2-478e-bfa1-4d0163b757ab","Type":"ContainerDied","Data":"e4d4a8bee9c7dcf8396e5948f71f6137a3411f0734980033354212e4c337252a"} Oct 30 06:16:15.573651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.572064 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4d4a8bee9c7dcf8396e5948f71f6137a3411f0734980033354212e4c337252a" Oct 30 06:16:15.576036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.575982 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.591903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.591566 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qqkc\" (UniqueName: \"kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc\") pod \"payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.591903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.591682 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.660057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.653775 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm"] Oct 30 06:16:15.723355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.723307 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qqkc\" (UniqueName: \"kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc\") pod \"payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.723957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.723933 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.725054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.724831 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.726461 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.726429 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bgbq\" (UniqueName: \"kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq\") pod \"payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:15.750006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.749966 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qqkc\" (UniqueName: \"kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc\") pod \"payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:15.768164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:15.768123 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:16.034339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.034294 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:16.092615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.092579 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24"] Oct 30 06:16:16.093512 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:16.093480 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda44482ec_6a3c_4c20_9d35_83b0e1aea0bb.slice/crio-526ddcb7e8266fba81021aa3631dd28e8e45da99236be7ffff4d6502f0b3df05 WatchSource:0}: Error finding container 526ddcb7e8266fba81021aa3631dd28e8e45da99236be7ffff4d6502f0b3df05: Status 404 returned error can't find the container with id 526ddcb7e8266fba81021aa3631dd28e8e45da99236be7ffff4d6502f0b3df05 Oct 30 06:16:16.400910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.400878 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm"] Oct 30 06:16:16.401162 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:16.401136 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aee4521_1565_4fd5_9e1f_e41227ad2ae7.slice/crio-88ae92b13eab135eb2f1bc65e2017a66664e0fb75c53ae7a6d884ee816527dc5 WatchSource:0}: Error finding container 88ae92b13eab135eb2f1bc65e2017a66664e0fb75c53ae7a6d884ee816527dc5: Status 404 returned error can't find the container with id 88ae92b13eab135eb2f1bc65e2017a66664e0fb75c53ae7a6d884ee816527dc5 Oct 30 06:16:16.574036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.573992 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" event={"ID":"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb","Type":"ContainerStarted","Data":"85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6"} Oct 30 06:16:16.574254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.574066 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:16.574254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.574087 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" event={"ID":"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb","Type":"ContainerStarted","Data":"526ddcb7e8266fba81021aa3631dd28e8e45da99236be7ffff4d6502f0b3df05"} Oct 30 06:16:16.576187 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.576154 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" event={"ID":"0aee4521-1565-4fd5-9e1f-e41227ad2ae7","Type":"ContainerStarted","Data":"e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed"} Oct 30 06:16:16.576391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.576209 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" event={"ID":"0aee4521-1565-4fd5-9e1f-e41227ad2ae7","Type":"ContainerStarted","Data":"88ae92b13eab135eb2f1bc65e2017a66664e0fb75c53ae7a6d884ee816527dc5"} Oct 30 06:16:16.576391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.576246 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:16.622586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.622507 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" podStartSLOduration=1.622489851 podStartE2EDuration="1.622489851s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:16.618235203 +0000 UTC m=+2683.193922260" watchObservedRunningTime="2026-10-30 06:16:16.622489851 +0000 UTC m=+2683.198176936" Oct 30 06:16:16.666630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:16.666466 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" podStartSLOduration=1.666448055 podStartE2EDuration="1.666448055s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:16.664286289 +0000 UTC m=+2683.239973344" watchObservedRunningTime="2026-10-30 06:16:16.666448055 +0000 UTC m=+2683.242135159" Oct 30 06:16:17.534045 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:17.533997 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:16:19.939547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:19.939486 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" podUID="1ac71497-b80b-4ab7-a3f1-c84cafc330c3" containerName="payload-processing" containerID="cri-o://030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe" gracePeriod=25 Oct 30 06:16:20.051395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.040241 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" podUID="3984d473-1aad-4f91-8268-24e38f11cf79" containerName="payload-pre-processing" containerID="cri-o://70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1" gracePeriod=25 Oct 30 06:16:20.393894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.393863 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 06:16:20.466091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.466055 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 06:16:20.574566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.574493 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b\" (UniqueName: \"kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b\") pod \"3984d473-1aad-4f91-8268-24e38f11cf79\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " Oct 30 06:16:20.574756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.574623 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd\" (UniqueName: \"kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd\") pod \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " Oct 30 06:16:20.574756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.574748 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume\") pod \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\" (UID: \"1ac71497-b80b-4ab7-a3f1-c84cafc330c3\") " Oct 30 06:16:20.574870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.574851 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume\") pod \"3984d473-1aad-4f91-8268-24e38f11cf79\" (UID: \"3984d473-1aad-4f91-8268-24e38f11cf79\") " Oct 30 06:16:20.575657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.575630 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume" pod "3984d473-1aad-4f91-8268-24e38f11cf79" (UID: "3984d473-1aad-4f91-8268-24e38f11cf79"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:20.576025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.576002 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume" pod "1ac71497-b80b-4ab7-a3f1-c84cafc330c3" (UID: "1ac71497-b80b-4ab7-a3f1-c84cafc330c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:20.578057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.578034 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd" pod "1ac71497-b80b-4ab7-a3f1-c84cafc330c3" (UID: "1ac71497-b80b-4ab7-a3f1-c84cafc330c3"). InnerVolumeSpecName "kube-api-access-dshtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:20.578644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.578617 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b" pod "3984d473-1aad-4f91-8268-24e38f11cf79" (UID: "3984d473-1aad-4f91-8268-24e38f11cf79"). InnerVolumeSpecName "kube-api-access-xcr8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:20.611823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.611459 5565 generic.go:356] "Generic (PLEG): container finished" podID="3984d473-1aad-4f91-8268-24e38f11cf79" containerID="70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1" exitCode=0 Oct 30 06:16:20.611823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.611497 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" event={"ID":"3984d473-1aad-4f91-8268-24e38f11cf79","Type":"ContainerDied","Data":"70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1"} Oct 30 06:16:20.611823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.611617 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" event={"ID":"3984d473-1aad-4f91-8268-24e38f11cf79","Type":"ContainerDied","Data":"622e2fe362486742928a93f9bc0496c86396ba4e62f2084d41b952f54a4224be"} Oct 30 06:16:20.611823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.611652 5565 scope.go:122] "RemoveContainer" containerID="70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1" Oct 30 06:16:20.611823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.611561 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b" Oct 30 06:16:20.614374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.614351 5565 generic.go:356] "Generic (PLEG): container finished" podID="1ac71497-b80b-4ab7-a3f1-c84cafc330c3" containerID="030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe" exitCode=0 Oct 30 06:16:20.614614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.614383 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" Oct 30 06:16:20.614828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.614401 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" event={"ID":"1ac71497-b80b-4ab7-a3f1-c84cafc330c3","Type":"ContainerDied","Data":"030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe"} Oct 30 06:16:20.615022 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.615003 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5" event={"ID":"1ac71497-b80b-4ab7-a3f1-c84cafc330c3","Type":"ContainerDied","Data":"24f02afe386b0463b65b6e265b2f9ba2a7bf2738ebd1c4bce95305f2dd0baa17"} Oct 30 06:16:20.626354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.626336 5565 scope.go:122] "RemoveContainer" containerID="70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1" Oct 30 06:16:20.626718 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:20.626689 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1\": container with ID starting with 70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1 not found: ID does not exist" containerID="70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1" Oct 30 06:16:20.626797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.626726 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1"} err="failed to get container status \"70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1\": rpc error: code = NotFound desc = could not find container \"70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1\": container with ID starting with 70b64e503defa8b35697acfed4475e4b4a8f479184308ba803556eec54bdeaf1 not found: ID does not exist" Oct 30 06:16:20.626797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.626750 5565 scope.go:122] "RemoveContainer" containerID="030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe" Oct 30 06:16:20.637881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.637860 5565 scope.go:122] "RemoveContainer" containerID="030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe" Oct 30 06:16:20.638207 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:20.638187 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe\": container with ID starting with 030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe not found: ID does not exist" containerID="030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe" Oct 30 06:16:20.638265 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.638217 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe"} err="failed to get container status \"030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe\": rpc error: code = NotFound desc = could not find container \"030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe\": container with ID starting with 030147b8cfb0e52e0038cc43fd76db5ae6171487e4bca2dc7cb97dc874f67efe not found: ID does not exist" Oct 30 06:16:20.678240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.678187 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:20.678240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.678226 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3984d473-1aad-4f91-8268-24e38f11cf79-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:20.678240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.678242 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xcr8b\" (UniqueName: \"kubernetes.io/projected/3984d473-1aad-4f91-8268-24e38f11cf79-kube-api-access-xcr8b\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:20.678497 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.678258 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dshtd\" (UniqueName: \"kubernetes.io/projected/1ac71497-b80b-4ab7-a3f1-c84cafc330c3-kube-api-access-dshtd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:20.686327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.686293 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5"] Oct 30 06:16:20.690012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.689985 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-698ab2-bcdfcf9db-zj2k5"] Oct 30 06:16:20.731981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.731902 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b"] Oct 30 06:16:20.737519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:20.737488 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-698ab2-54974d5759-hqf2b"] Oct 30 06:16:21.693551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:21.680245 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ac71497-b80b-4ab7-a3f1-c84cafc330c3" path="/var/lib/kubelet/pods/1ac71497-b80b-4ab7-a3f1-c84cafc330c3/volumes" Oct 30 06:16:21.693551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:21.680957 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3984d473-1aad-4f91-8268-24e38f11cf79" path="/var/lib/kubelet/pods/3984d473-1aad-4f91-8268-24e38f11cf79/volumes" Oct 30 06:16:22.639487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.639450 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx"] Oct 30 06:16:22.640230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.640213 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3984d473-1aad-4f91-8268-24e38f11cf79" containerName="payload-pre-processing" Oct 30 06:16:22.640295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.640230 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3984d473-1aad-4f91-8268-24e38f11cf79" containerName="payload-pre-processing" Oct 30 06:16:22.640295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.640246 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1ac71497-b80b-4ab7-a3f1-c84cafc330c3" containerName="payload-processing" Oct 30 06:16:22.640295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.640251 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ac71497-b80b-4ab7-a3f1-c84cafc330c3" containerName="payload-processing" Oct 30 06:16:22.640413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.640389 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3984d473-1aad-4f91-8268-24e38f11cf79" containerName="payload-pre-processing" Oct 30 06:16:22.640413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.640402 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ac71497-b80b-4ab7-a3f1-c84cafc330c3" containerName="payload-processing" Oct 30 06:16:22.645754 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.645729 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.669069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.669039 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx"] Oct 30 06:16:22.678331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.678297 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:22.678517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.678387 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:16:22.720713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.720672 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.721221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.720862 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5tp7\" (UniqueName: \"kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.721221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.720959 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.822081 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.822042 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.822279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.822171 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5tp7\" (UniqueName: \"kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.822279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.822220 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.822759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.822734 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.824511 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.824488 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.847771 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.847740 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5tp7\" (UniqueName: \"kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7\") pod \"maas-api-revoke-sub-d5c62456-qdfsx\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:22.961227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:22.961130 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:23.273604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:23.273569 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx"] Oct 30 06:16:23.274819 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:23.274793 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3cb5b4_4c8a_4098_a3d9_538bfcf69964.slice/crio-3db56143b2edd3cd464f5f20a23d3137480ada97f40e26ab115d593dadf54edc WatchSource:0}: Error finding container 3db56143b2edd3cd464f5f20a23d3137480ada97f40e26ab115d593dadf54edc: Status 404 returned error can't find the container with id 3db56143b2edd3cd464f5f20a23d3137480ada97f40e26ab115d593dadf54edc Oct 30 06:16:23.707496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:23.707461 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" event={"ID":"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964","Type":"ContainerStarted","Data":"18ee9a7b8d4b2fbadbd31de4229b7cf39850bd133824ab179ff1ee0b6f0fbabf"} Oct 30 06:16:23.707696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:23.707513 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" event={"ID":"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964","Type":"ContainerStarted","Data":"3db56143b2edd3cd464f5f20a23d3137480ada97f40e26ab115d593dadf54edc"} Oct 30 06:16:23.754181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:23.754127 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" podStartSLOduration=1.754111433 podStartE2EDuration="1.754111433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:23.7531387 +0000 UTC m=+2690.328825755" watchObservedRunningTime="2026-10-30 06:16:23.754111433 +0000 UTC m=+2690.329798486" Oct 30 06:16:24.376869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.376821 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" containerID="cri-o://2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243" gracePeriod=45 Oct 30 06:16:24.382603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.382575 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" containerID="cri-o://cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9" gracePeriod=45 Oct 30 06:16:24.692583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.692518 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 06:16:24.696560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.696519 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 06:16:24.716588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.716519 5565 generic.go:356] "Generic (PLEG): container finished" podID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerID="2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243" exitCode=0 Oct 30 06:16:24.716804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.716608 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" Oct 30 06:16:24.716804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.716557 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" event={"ID":"19085093-07d1-4cf4-813f-04f6e159b7dc","Type":"ContainerDied","Data":"2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243"} Oct 30 06:16:24.716804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.716685 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg" event={"ID":"19085093-07d1-4cf4-813f-04f6e159b7dc","Type":"ContainerDied","Data":"9b34af30277e2c9df5eb0753858a0ba3e30f41631677fcede313f486445cb0df"} Oct 30 06:16:24.716804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.716710 5565 scope.go:122] "RemoveContainer" containerID="2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243" Oct 30 06:16:24.718676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.718648 5565 generic.go:356] "Generic (PLEG): container finished" podID="3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" containerID="18ee9a7b8d4b2fbadbd31de4229b7cf39850bd133824ab179ff1ee0b6f0fbabf" exitCode=0 Oct 30 06:16:24.718841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.718810 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" event={"ID":"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964","Type":"ContainerDied","Data":"18ee9a7b8d4b2fbadbd31de4229b7cf39850bd133824ab179ff1ee0b6f0fbabf"} Oct 30 06:16:24.720384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.720357 5565 generic.go:356] "Generic (PLEG): container finished" podID="7759d770-f847-42d7-a258-20dedae07d2a" containerID="cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9" exitCode=0 Oct 30 06:16:24.720497 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.720395 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" Oct 30 06:16:24.720497 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.720403 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" event={"ID":"7759d770-f847-42d7-a258-20dedae07d2a","Type":"ContainerDied","Data":"cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9"} Oct 30 06:16:24.720497 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.720434 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd" event={"ID":"7759d770-f847-42d7-a258-20dedae07d2a","Type":"ContainerDied","Data":"fe45b0dc835755b355a236291076d2f903b389d9bd9b8def81796a54c98a0ed0"} Oct 30 06:16:24.728897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.728873 5565 scope.go:122] "RemoveContainer" containerID="2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243" Oct 30 06:16:24.729263 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:24.729236 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243\": container with ID starting with 2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243 not found: ID does not exist" containerID="2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243" Oct 30 06:16:24.729339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.729272 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243"} err="failed to get container status \"2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243\": rpc error: code = NotFound desc = could not find container \"2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243\": container with ID starting with 2ee6420bcfa20a5e50af099c0d9c412a4f5ef8f6ac675313147c8467f8c8c243 not found: ID does not exist" Oct 30 06:16:24.729339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.729291 5565 scope.go:122] "RemoveContainer" containerID="cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9" Oct 30 06:16:24.742168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.742142 5565 scope.go:122] "RemoveContainer" containerID="cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9" Oct 30 06:16:24.742676 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:24.742519 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9\": container with ID starting with cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9 not found: ID does not exist" containerID="cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9" Oct 30 06:16:24.742797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.742689 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9"} err="failed to get container status \"cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9\": rpc error: code = NotFound desc = could not find container \"cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9\": container with ID starting with cec961d1668467f63a6a4ef04f555e875d6cfade75510184da85d0eed50a24a9 not found: ID does not exist" Oct 30 06:16:24.751122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751086 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm\") pod \"19085093-07d1-4cf4-813f-04f6e159b7dc\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " Oct 30 06:16:24.751308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751203 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs\" (UniqueName: \"kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs\") pod \"19085093-07d1-4cf4-813f-04f6e159b7dc\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " Oct 30 06:16:24.751308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751288 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm\") pod \"7759d770-f847-42d7-a258-20dedae07d2a\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " Oct 30 06:16:24.751592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751377 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache\") pod \"19085093-07d1-4cf4-813f-04f6e159b7dc\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " Oct 30 06:16:24.751592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751416 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home\") pod \"19085093-07d1-4cf4-813f-04f6e159b7dc\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " Oct 30 06:16:24.751592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751466 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir\") pod \"7759d770-f847-42d7-a258-20dedae07d2a\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " Oct 30 06:16:24.751811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751777 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir" pod "7759d770-f847-42d7-a258-20dedae07d2a" (UID: "7759d770-f847-42d7-a258-20dedae07d2a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.751944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.751920 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache" pod "19085093-07d1-4cf4-813f-04f6e159b7dc" (UID: "19085093-07d1-4cf4-813f-04f6e159b7dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.752199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.752178 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2\" (UniqueName: \"kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2\") pod \"7759d770-f847-42d7-a258-20dedae07d2a\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " Oct 30 06:16:24.752343 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.752309 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home" pod "19085093-07d1-4cf4-813f-04f6e159b7dc" (UID: "19085093-07d1-4cf4-813f-04f6e159b7dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.752814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.752792 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5\" (UniqueName: \"kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5\") pod \"19085093-07d1-4cf4-813f-04f6e159b7dc\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " Oct 30 06:16:24.752934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.752906 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs\" (UniqueName: \"kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs\") pod \"7759d770-f847-42d7-a258-20dedae07d2a\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " Oct 30 06:16:24.753017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.753003 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home\") pod \"7759d770-f847-42d7-a258-20dedae07d2a\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " Oct 30 06:16:24.753084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.753047 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir\") pod \"19085093-07d1-4cf4-813f-04f6e159b7dc\" (UID: \"19085093-07d1-4cf4-813f-04f6e159b7dc\") " Oct 30 06:16:24.753146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.753105 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache\") pod \"7759d770-f847-42d7-a258-20dedae07d2a\" (UID: \"7759d770-f847-42d7-a258-20dedae07d2a\") " Oct 30 06:16:24.753776 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.753583 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home" pod "7759d770-f847-42d7-a258-20dedae07d2a" (UID: "7759d770-f847-42d7-a258-20dedae07d2a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.753964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.753929 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir" pod "19085093-07d1-4cf4-813f-04f6e159b7dc" (UID: "19085093-07d1-4cf4-813f-04f6e159b7dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754252 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache" pod "7759d770-f847-42d7-a258-20dedae07d2a" (UID: "7759d770-f847-42d7-a258-20dedae07d2a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754638 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754660 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754674 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754688 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754701 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.754747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.754713 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.756678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.756640 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs" pod "19085093-07d1-4cf4-813f-04f6e159b7dc" (UID: "19085093-07d1-4cf4-813f-04f6e159b7dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:24.762985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.762921 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm" pod "7759d770-f847-42d7-a258-20dedae07d2a" (UID: "7759d770-f847-42d7-a258-20dedae07d2a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.763162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.762998 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5" pod "19085093-07d1-4cf4-813f-04f6e159b7dc" (UID: "19085093-07d1-4cf4-813f-04f6e159b7dc"). InnerVolumeSpecName "kube-api-access-vlzk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:24.763162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.763046 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2" pod "7759d770-f847-42d7-a258-20dedae07d2a" (UID: "7759d770-f847-42d7-a258-20dedae07d2a"). InnerVolumeSpecName "kube-api-access-wxbk2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:24.763434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.763410 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm" pod "19085093-07d1-4cf4-813f-04f6e159b7dc" (UID: "19085093-07d1-4cf4-813f-04f6e159b7dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:16:24.765407 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.765374 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs" pod "7759d770-f847-42d7-a258-20dedae07d2a" (UID: "7759d770-f847-42d7-a258-20dedae07d2a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:24.856541 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.856499 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7759d770-f847-42d7-a258-20dedae07d2a-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.856738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.856570 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxbk2\" (UniqueName: \"kubernetes.io/projected/7759d770-f847-42d7-a258-20dedae07d2a-kube-api-access-wxbk2\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.856738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.856583 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vlzk5\" (UniqueName: \"kubernetes.io/projected/19085093-07d1-4cf4-813f-04f6e159b7dc-kube-api-access-vlzk5\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.856738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.856593 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7759d770-f847-42d7-a258-20dedae07d2a-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.856738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.856602 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19085093-07d1-4cf4-813f-04f6e159b7dc-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:24.856738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:24.856609 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19085093-07d1-4cf4-813f-04f6e159b7dc-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.068586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.068522 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg"] Oct 30 06:16:25.072020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.071999 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/distinct-w4-698ab2-kserve-57d58c647f-mxbzg"] Oct 30 06:16:25.109023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.108955 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd"] Oct 30 06:16:25.115064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.115036 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-698ab2/premium-w4-698ab2-kserve-7598ff888d-44gwd"] Oct 30 06:16:25.568789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.568757 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" path="/var/lib/kubelet/pods/19085093-07d1-4cf4-813f-04f6e159b7dc/volumes" Oct 30 06:16:25.569196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.569181 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7759d770-f847-42d7-a258-20dedae07d2a" path="/var/lib/kubelet/pods/7759d770-f847-42d7-a258-20dedae07d2a/volumes" Oct 30 06:16:25.900273 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.900241 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:25.973073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.973040 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7\" (UniqueName: \"kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7\") pod \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " Oct 30 06:16:25.973258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.973154 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca\") pod \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " Oct 30 06:16:25.973258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.973241 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca\") pod \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\" (UID: \"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964\") " Oct 30 06:16:25.973683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.973652 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca" pod "3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" (UID: "3c3cb5b4-4c8a-4098-a3d9-538bfcf69964"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:25.974096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.974068 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:25.975410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.975384 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7" pod "3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" (UID: "3c3cb5b4-4c8a-4098-a3d9-538bfcf69964"). InnerVolumeSpecName "kube-api-access-k5tp7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:25.975695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:25.975674 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca" pod "3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" (UID: "3c3cb5b4-4c8a-4098-a3d9-538bfcf69964"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:26.075910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.075879 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k5tp7\" (UniqueName: \"kubernetes.io/projected/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-kube-api-access-k5tp7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.075910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.075908 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:26.682308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.682270 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf"] Oct 30 06:16:26.683138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683114 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" Oct 30 06:16:26.683138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683134 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683147 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" containerName="revoke-keys" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683152 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" containerName="revoke-keys" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683166 5565 cpu_manager.go:414] "containerMap: removing container" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683171 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683277 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7759d770-f847-42d7-a258-20dedae07d2a" containerName="main" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683285 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" containerName="revoke-keys" Oct 30 06:16:26.683350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.683295 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="19085093-07d1-4cf4-813f-04f6e159b7dc" containerName="main" Oct 30 06:16:26.687519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.687499 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.724901 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.724868 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf"] Oct 30 06:16:26.741159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.741135 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" Oct 30 06:16:26.741317 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.741173 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx" event={"ID":"3c3cb5b4-4c8a-4098-a3d9-538bfcf69964","Type":"ContainerDied","Data":"3db56143b2edd3cd464f5f20a23d3137480ada97f40e26ab115d593dadf54edc"} Oct 30 06:16:26.741317 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.741222 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3db56143b2edd3cd464f5f20a23d3137480ada97f40e26ab115d593dadf54edc" Oct 30 06:16:26.785207 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785176 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785262 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785370 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785420 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785507 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785563 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785596 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67zdj\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785652 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.785797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.785781 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.886887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.886855 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.886945 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887017 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887131 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887216 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887198 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887241 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887271 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887299 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887314 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887366 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67zdj\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887372 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887785 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.887811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887799 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.888009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.887986 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.889314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.889283 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.889713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.889693 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.913708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.913664 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67zdj\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:26.914052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:26.913785 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token\") pod \"e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:27.002570 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.002520 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:27.327860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.327830 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf"] Oct 30 06:16:27.328479 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:27.328451 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3c91cec_c5fd_42a0_9135_3210a8261b02.slice/crio-aac88cf9fafd019586c540588d35be72c6954b0baa4156c588a14449e3fb1985 WatchSource:0}: Error finding container aac88cf9fafd019586c540588d35be72c6954b0baa4156c588a14449e3fb1985: Status 404 returned error can't find the container with id aac88cf9fafd019586c540588d35be72c6954b0baa4156c588a14449e3fb1985 Oct 30 06:16:27.331415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.331382 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:27.331484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.331464 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:27.331547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.331504 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:27.585409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.585337 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:16:27.585564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.585419 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:16:27.751854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.751817 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" event={"ID":"d3c91cec-c5fd-42a0-9135-3210a8261b02","Type":"ContainerStarted","Data":"71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78"} Oct 30 06:16:27.752043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.751873 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" event={"ID":"d3c91cec-c5fd-42a0-9135-3210a8261b02","Type":"ContainerStarted","Data":"aac88cf9fafd019586c540588d35be72c6954b0baa4156c588a14449e3fb1985"} Oct 30 06:16:27.847748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:27.847628 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podStartSLOduration=1.847608406 podStartE2EDuration="1.847608406s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:27.829485116 +0000 UTC m=+2694.405172170" watchObservedRunningTime="2026-10-30 06:16:27.847608406 +0000 UTC m=+2694.423295461" Oct 30 06:16:28.003523 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:28.003491 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:28.005255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:28.005227 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:28.005398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:28.005278 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:29.002911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:29.002872 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:29.003101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:29.002928 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:30.003892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.003859 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:30.004291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.003911 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:30.730729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.730694 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn"] Oct 30 06:16:30.735900 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.735872 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.746448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.746416 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh"] Oct 30 06:16:30.774799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.774763 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.809247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.804948 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-de2bf2-openshift-default-dockercfg-s762q\"" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.817851 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.817927 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.817990 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818075 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818133 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818196 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818222 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtxjm\" (UniqueName: \"kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818258 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818299 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818338 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818435 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfz6p\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.830821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.818484 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.833567 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.833466 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn"] Oct 30 06:16:30.868068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.864822 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:16:30.868068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.865004 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929234 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929334 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929367 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtxjm\" (UniqueName: \"kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929400 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929441 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929478 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929588 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfz6p\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929635 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929694 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929752 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929823 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.932140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.929938 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:30.957560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:30.955515 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.017699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.004399 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:31.017699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.004459 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:31.025615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.025192 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:31.025969 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.025849 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.027898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.026927 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.030069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.028291 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.030069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.028781 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.030069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.029805 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.030880 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.030788 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.032037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.032012 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:31.035382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.035349 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh"] Oct 30 06:16:31.097322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.096185 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.100778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.100712 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtxjm\" (UniqueName: \"kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm\") pod \"maas-api-revoke-sub-4b66648f-88nsh\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:31.101559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.101489 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfz6p\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p\") pod \"e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.119172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.119132 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:31.176553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.173806 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:31.500287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.500247 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:31.500428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.500344 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:31.500428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.500390 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:31.506652 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.506620 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn"] Oct 30 06:16:31.792552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:31.790638 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh"] Oct 30 06:16:32.003985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.003941 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:32.004165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.004007 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:32.041979 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.041926 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" event={"ID":"ce571bd4-e31b-41e4-8e3b-3a3459e765f1","Type":"ContainerStarted","Data":"a4d4ac5f8d1d7131c6f73131115ce04c344774f816852c01dc3be45159e0ff0b"} Oct 30 06:16:32.043976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.043938 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" event={"ID":"4cb2b128-3deb-4ef7-b632-019fac593c36","Type":"ContainerStarted","Data":"05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42"} Oct 30 06:16:32.044159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.043995 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" event={"ID":"4cb2b128-3deb-4ef7-b632-019fac593c36","Type":"ContainerStarted","Data":"d7288a3267762068b1dbb85032a9fcb2f1a649ea5a3bfc69b0c142e796ca7f73"} Oct 30 06:16:32.111454 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.111376 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podStartSLOduration=2.111354833 podStartE2EDuration="2.111354833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:32.108813509 +0000 UTC m=+2698.684500564" watchObservedRunningTime="2026-10-30 06:16:32.111354833 +0000 UTC m=+2698.687041888" Oct 30 06:16:32.119775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.119736 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:32.122074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.122039 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:32.122231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.122100 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:32.886299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.886238 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x"] Oct 30 06:16:32.937645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.935114 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:32.938822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:32.938771 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x"] Oct 30 06:16:33.013287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.013256 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:33.013441 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.013308 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:33.042131 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.042056 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kd77\" (UniqueName: \"kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.042795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.042411 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.042795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.042523 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.117608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.117572 5565 generic.go:356] "Generic (PLEG): container finished" podID="ce571bd4-e31b-41e4-8e3b-3a3459e765f1" containerID="1fcfa13562e112254a168829a91706439168054bf1f0d01444025b44b9f0aafe" exitCode=0 Oct 30 06:16:33.118636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.117810 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" event={"ID":"ce571bd4-e31b-41e4-8e3b-3a3459e765f1","Type":"ContainerDied","Data":"1fcfa13562e112254a168829a91706439168054bf1f0d01444025b44b9f0aafe"} Oct 30 06:16:33.120172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.120148 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:33.120352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.120319 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:33.143757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.143652 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd77\" (UniqueName: \"kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.143962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.143856 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.143962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.143921 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.144950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.144928 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.146566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.146484 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.349258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.347649 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd77\" (UniqueName: \"kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77\") pod \"maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:33.624731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:33.624674 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:34.017010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.016965 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.017206 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.017032 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.9:15021/healthz/ready\": dial tcp 10.134.1.9:15021: connect: connection refused" Oct 30 06:16:34.120245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.120203 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:34.120840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.120268 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:34.127797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.126789 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf"] Oct 30 06:16:34.129639 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:34.129022 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5672b652_40ea_41dd_ba82_6bcc491caea5.slice/crio-a4b95f7c6e9240573674fab1c98a2a32f8892df9e6bcb5299dcc66a662898ed7 WatchSource:0}: Error finding container a4b95f7c6e9240573674fab1c98a2a32f8892df9e6bcb5299dcc66a662898ed7: Status 404 returned error can't find the container with id a4b95f7c6e9240573674fab1c98a2a32f8892df9e6bcb5299dcc66a662898ed7 Oct 30 06:16:34.132938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.132913 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-edfa701c-pssmf"] Oct 30 06:16:34.140618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.139347 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x"] Oct 30 06:16:34.620408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.620380 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:34.728558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.727675 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca\") pod \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " Oct 30 06:16:34.728558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.727928 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm\" (UniqueName: \"kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm\") pod \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " Oct 30 06:16:34.728558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.728010 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca\") pod \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\" (UID: \"ce571bd4-e31b-41e4-8e3b-3a3459e765f1\") " Oct 30 06:16:34.729622 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.729543 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca" pod "ce571bd4-e31b-41e4-8e3b-3a3459e765f1" (UID: "ce571bd4-e31b-41e4-8e3b-3a3459e765f1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:34.733365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.731746 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca" pod "ce571bd4-e31b-41e4-8e3b-3a3459e765f1" (UID: "ce571bd4-e31b-41e4-8e3b-3a3459e765f1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:34.790548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.790483 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm" pod "ce571bd4-e31b-41e4-8e3b-3a3459e765f1" (UID: "ce571bd4-e31b-41e4-8e3b-3a3459e765f1"). InnerVolumeSpecName "kube-api-access-dtxjm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:34.880620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.879925 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dtxjm\" (UniqueName: \"kubernetes.io/projected/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-kube-api-access-dtxjm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:34.880620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.879967 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:34.880620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:34.879984 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce571bd4-e31b-41e4-8e3b-3a3459e765f1-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:35.015594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.014389 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:35.015803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.015734 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:35.016513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.016477 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:16:35.032183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.032140 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq"] Oct 30 06:16:35.040333 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.040288 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7b1f55a5-s2wmq"] Oct 30 06:16:35.120628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.120590 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:35.121052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.120643 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:35.146481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.146383 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" event={"ID":"5672b652-40ea-41dd-ba82-6bcc491caea5","Type":"ContainerStarted","Data":"4b53a52da9e3e5aab117a459a76e4fe87e851a5c6bc66b8d971818bac544f2bf"} Oct 30 06:16:35.146481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.146452 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" event={"ID":"5672b652-40ea-41dd-ba82-6bcc491caea5","Type":"ContainerStarted","Data":"a4b95f7c6e9240573674fab1c98a2a32f8892df9e6bcb5299dcc66a662898ed7"} Oct 30 06:16:35.148395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.148359 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" event={"ID":"ce571bd4-e31b-41e4-8e3b-3a3459e765f1","Type":"ContainerDied","Data":"a4d4ac5f8d1d7131c6f73131115ce04c344774f816852c01dc3be45159e0ff0b"} Oct 30 06:16:35.148395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.148418 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4d4ac5f8d1d7131c6f73131115ce04c344774f816852c01dc3be45159e0ff0b" Oct 30 06:16:35.148681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.148374 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh" Oct 30 06:16:35.216016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.215836 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" podStartSLOduration=3.215817973 podStartE2EDuration="3.215817973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:35.208840976 +0000 UTC m=+2701.784528029" watchObservedRunningTime="2026-10-30 06:16:35.215817973 +0000 UTC m=+2701.791505029" Oct 30 06:16:35.595233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.595193 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="864ed9ac-7d96-4c1b-86d5-c1c27bfa17af" path="/var/lib/kubelet/pods/864ed9ac-7d96-4c1b-86d5-c1c27bfa17af/volumes" Oct 30 06:16:35.595780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:35.595764 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca947cce-3800-4ed7-818f-5495b308430d" path="/var/lib/kubelet/pods/ca947cce-3800-4ed7-818f-5495b308430d/volumes" Oct 30 06:16:36.120514 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:36.120476 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:36.120514 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:36.120545 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:36.179735 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:36.179702 5565 generic.go:356] "Generic (PLEG): container finished" podID="5672b652-40ea-41dd-ba82-6bcc491caea5" containerID="4b53a52da9e3e5aab117a459a76e4fe87e851a5c6bc66b8d971818bac544f2bf" exitCode=0 Oct 30 06:16:36.180633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:36.180600 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" event={"ID":"5672b652-40ea-41dd-ba82-6bcc491caea5","Type":"ContainerDied","Data":"4b53a52da9e3e5aab117a459a76e4fe87e851a5c6bc66b8d971818bac544f2bf"} Oct 30 06:16:37.120381 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.120341 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:37.120587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.120417 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:37.596080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.596048 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:37.684442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.684399 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca\") pod \"5672b652-40ea-41dd-ba82-6bcc491caea5\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " Oct 30 06:16:37.684740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.684572 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca\") pod \"5672b652-40ea-41dd-ba82-6bcc491caea5\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " Oct 30 06:16:37.684740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.684665 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77\" (UniqueName: \"kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77\") pod \"5672b652-40ea-41dd-ba82-6bcc491caea5\" (UID: \"5672b652-40ea-41dd-ba82-6bcc491caea5\") " Oct 30 06:16:37.685257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.685220 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca" pod "5672b652-40ea-41dd-ba82-6bcc491caea5" (UID: "5672b652-40ea-41dd-ba82-6bcc491caea5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:37.685626 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.685601 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:37.688144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.688113 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77" pod "5672b652-40ea-41dd-ba82-6bcc491caea5" (UID: "5672b652-40ea-41dd-ba82-6bcc491caea5"). InnerVolumeSpecName "kube-api-access-9kd77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:37.689760 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.689730 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca" pod "5672b652-40ea-41dd-ba82-6bcc491caea5" (UID: "5672b652-40ea-41dd-ba82-6bcc491caea5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:16:37.786971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.786878 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5672b652-40ea-41dd-ba82-6bcc491caea5-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:37.786971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:37.786915 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kd77\" (UniqueName: \"kubernetes.io/projected/5672b652-40ea-41dd-ba82-6bcc491caea5-kube-api-access-9kd77\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:38.120455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.120362 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" start-of-body= Oct 30 06:16:38.120455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.120412 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.10:15021/healthz/ready\": dial tcp 10.134.1.10:15021: connect: connection refused" Oct 30 06:16:38.198543 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.198488 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" event={"ID":"5672b652-40ea-41dd-ba82-6bcc491caea5","Type":"ContainerDied","Data":"a4b95f7c6e9240573674fab1c98a2a32f8892df9e6bcb5299dcc66a662898ed7"} Oct 30 06:16:38.198543 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.198514 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x" Oct 30 06:16:38.198774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.198586 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b95f7c6e9240573674fab1c98a2a32f8892df9e6bcb5299dcc66a662898ed7" Oct 30 06:16:38.658441 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.658399 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8"] Oct 30 06:16:38.662299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.662253 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ce571bd4-e31b-41e4-8e3b-3a3459e765f1" containerName="revoke-keys" Oct 30 06:16:38.662299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.662292 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce571bd4-e31b-41e4-8e3b-3a3459e765f1" containerName="revoke-keys" Oct 30 06:16:38.662510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.662331 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5672b652-40ea-41dd-ba82-6bcc491caea5" containerName="revoke-keys" Oct 30 06:16:38.662510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.662339 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5672b652-40ea-41dd-ba82-6bcc491caea5" containerName="revoke-keys" Oct 30 06:16:38.662653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.662576 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce571bd4-e31b-41e4-8e3b-3a3459e765f1" containerName="revoke-keys" Oct 30 06:16:38.662653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.662595 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5672b652-40ea-41dd-ba82-6bcc491caea5" containerName="revoke-keys" Oct 30 06:16:38.667920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.667861 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.681560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.681508 5565 scope.go:122] "RemoveContainer" containerID="a3507f07071327f13f89cd7a839e79b9ead93a871b14112f700736f680d66553" Oct 30 06:16:38.691517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.691488 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-07b278-openshift-default-dockercfg-srhdh\"" Oct 30 06:16:38.697622 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.697590 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8"] Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.826652 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2nb\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.826745 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.826802 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.826845 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.826925 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.826988 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.827077 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.827270 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.831503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.827304 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929047 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.928956 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2nb\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929047 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929008 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929056 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929094 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929159 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929213 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929286 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929401 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929437 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929570 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929707 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929745 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.929797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.929790 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.930063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.930034 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.931663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.931642 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.932012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.931992 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.962020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.961987 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.962020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.962018 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2nb\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb\") pod \"e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:38.997050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:38.997009 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:39.128521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.128483 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:39.129169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.129142 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:39.131474 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.131445 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:16:39.336499 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:39.336457 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod121ba31d_7c52_4d21_bac3_fc3848d2a8f4.slice/crio-1ff2411f68ff89f3c5b96d2049d43709bed67d555ca3f1004afcb82f9e818332 WatchSource:0}: Error finding container 1ff2411f68ff89f3c5b96d2049d43709bed67d555ca3f1004afcb82f9e818332: Status 404 returned error can't find the container with id 1ff2411f68ff89f3c5b96d2049d43709bed67d555ca3f1004afcb82f9e818332 Oct 30 06:16:39.340807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.340771 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8"] Oct 30 06:16:39.350548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.350492 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:39.350680 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.350581 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:39.350680 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:39.350614 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:16:40.217860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:40.217815 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" event={"ID":"121ba31d-7c52-4d21-bac3-fc3848d2a8f4","Type":"ContainerStarted","Data":"073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403"} Oct 30 06:16:40.218335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:40.217888 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" event={"ID":"121ba31d-7c52-4d21-bac3-fc3848d2a8f4","Type":"ContainerStarted","Data":"1ff2411f68ff89f3c5b96d2049d43709bed67d555ca3f1004afcb82f9e818332"} Oct 30 06:16:40.272680 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:40.272610 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podStartSLOduration=2.272587946 podStartE2EDuration="2.272587946s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:40.269111675 +0000 UTC m=+2706.844798732" watchObservedRunningTime="2026-10-30 06:16:40.272587946 +0000 UTC m=+2706.848275001" Oct 30 06:16:40.997594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:40.997553 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:41.004025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:41.003994 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:41.226657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:41.226630 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:41.227588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:41.227568 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:16:44.505689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.505629 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm"] Oct 30 06:16:44.512182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.512147 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.530359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.529991 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm"] Oct 30 06:16:44.610555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.605178 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07b278-dockercfg-92k4m\"" Oct 30 06:16:44.610555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.605303 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07b278\"" Oct 30 06:16:44.631550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.630108 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qpkq\" (UniqueName: \"kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq\") pod \"payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.631550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.630305 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.771695 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.771586 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qpkq\" (UniqueName: \"kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq\") pod \"payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.771893 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.771753 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.801621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.772744 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.801976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.800695 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qpkq\" (UniqueName: \"kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq\") pod \"payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:44.842280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:44.842233 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:45.032266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:45.032170 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4"] Oct 30 06:16:45.036294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:45.036256 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c5663160-snfr4"] Oct 30 06:16:45.166479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:45.166437 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm"] Oct 30 06:16:45.166871 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:16:45.166847 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6849eb2_8b7b_4a87_a0df_e39d1903e41c.slice/crio-9b80b8bd7d253870b2fae77766439e8b89cf56883218eaf5d007dc7e539e35cb WatchSource:0}: Error finding container 9b80b8bd7d253870b2fae77766439e8b89cf56883218eaf5d007dc7e539e35cb: Status 404 returned error can't find the container with id 9b80b8bd7d253870b2fae77766439e8b89cf56883218eaf5d007dc7e539e35cb Oct 30 06:16:45.263757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:45.263717 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" event={"ID":"f6849eb2-8b7b-4a87-a0df-e39d1903e41c","Type":"ContainerStarted","Data":"9b80b8bd7d253870b2fae77766439e8b89cf56883218eaf5d007dc7e539e35cb"} Oct 30 06:16:45.571207 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:45.571166 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a48e662a-4211-4b5c-bb5c-3ee182ef7175" path="/var/lib/kubelet/pods/a48e662a-4211-4b5c-bb5c-3ee182ef7175/volumes" Oct 30 06:16:46.325080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:46.325036 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" event={"ID":"f6849eb2-8b7b-4a87-a0df-e39d1903e41c","Type":"ContainerStarted","Data":"f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db"} Oct 30 06:16:46.325324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:46.325144 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:46.376258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:46.376190 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" podStartSLOduration=2.376171122 podStartE2EDuration="2.376171122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:16:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:16:46.372608227 +0000 UTC m=+2712.948295283" watchObservedRunningTime="2026-10-30 06:16:46.376171122 +0000 UTC m=+2712.951858176" Oct 30 06:16:47.532862 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:16:47.532820 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:16:52.257707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:52.257670 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4"] Oct 30 06:16:57.258092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.258042 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" podUID="cc92ace7-b66c-41c0-95e7-736a244b421f" containerName="payload-pre-processing" containerID="cri-o://c9d800761dc8083b9aac83b600febaccb15bf113ac57552704f86b97bad5ca0b" gracePeriod=25 Oct 30 06:16:57.336301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.336271 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:16:57.456818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.456775 5565 generic.go:356] "Generic (PLEG): container finished" podID="cc92ace7-b66c-41c0-95e7-736a244b421f" containerID="c9d800761dc8083b9aac83b600febaccb15bf113ac57552704f86b97bad5ca0b" exitCode=0 Oct 30 06:16:57.457121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.457095 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" event={"ID":"cc92ace7-b66c-41c0-95e7-736a244b421f","Type":"ContainerDied","Data":"c9d800761dc8083b9aac83b600febaccb15bf113ac57552704f86b97bad5ca0b"} Oct 30 06:16:57.555467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.555422 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:16:57.709852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.709814 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4\" (UniqueName: \"kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4\") pod \"cc92ace7-b66c-41c0-95e7-736a244b421f\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " Oct 30 06:16:57.710043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.709934 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume\") pod \"cc92ace7-b66c-41c0-95e7-736a244b421f\" (UID: \"cc92ace7-b66c-41c0-95e7-736a244b421f\") " Oct 30 06:16:57.710288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.710259 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume" pod "cc92ace7-b66c-41c0-95e7-736a244b421f" (UID: "cc92ace7-b66c-41c0-95e7-736a244b421f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:16:57.710831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.710810 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc92ace7-b66c-41c0-95e7-736a244b421f-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:57.712066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.712042 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4" pod "cc92ace7-b66c-41c0-95e7-736a244b421f" (UID: "cc92ace7-b66c-41c0-95e7-736a244b421f"). InnerVolumeSpecName "kube-api-access-g49z4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:16:57.811778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:57.811718 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g49z4\" (UniqueName: \"kubernetes.io/projected/cc92ace7-b66c-41c0-95e7-736a244b421f-kube-api-access-g49z4\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:16:58.466200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:58.466167 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" event={"ID":"cc92ace7-b66c-41c0-95e7-736a244b421f","Type":"ContainerDied","Data":"3c6db571836387d5293c4577ca70bf293fb97664d36a0b40d9c0cd9bbd4e8901"} Oct 30 06:16:58.466200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:58.466193 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4" Oct 30 06:16:58.466775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:58.466237 5565 scope.go:122] "RemoveContainer" containerID="c9d800761dc8083b9aac83b600febaccb15bf113ac57552704f86b97bad5ca0b" Oct 30 06:16:58.527649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:58.527618 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4"] Oct 30 06:16:58.541594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:58.541567 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6bf867794c-bb8c4"] Oct 30 06:16:59.568694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:16:59.568653 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc92ace7-b66c-41c0-95e7-736a244b421f" path="/var/lib/kubelet/pods/cc92ace7-b66c-41c0-95e7-736a244b421f/volumes" Oct 30 06:17:14.175547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:14.175152 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:17:17.533455 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:17:17.533414 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:17:22.025243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.025202 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk"] Oct 30 06:17:22.040132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.040097 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cc92ace7-b66c-41c0-95e7-736a244b421f" containerName="payload-pre-processing" Oct 30 06:17:22.040342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.040315 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc92ace7-b66c-41c0-95e7-736a244b421f" containerName="payload-pre-processing" Oct 30 06:17:22.040772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.040750 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc92ace7-b66c-41c0-95e7-736a244b421f" containerName="payload-pre-processing" Oct 30 06:17:22.067785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.067752 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk"] Oct 30 06:17:22.068190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.068120 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.090691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.087809 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.090691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.088055 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zhjv\" (UniqueName: \"kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.106030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.105997 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5a591d-dockercfg-jkhn8\"" Oct 30 06:17:22.106603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.106391 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5a591d\"" Oct 30 06:17:22.189325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.189274 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7"] Oct 30 06:17:22.196771 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.196731 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zhjv\" (UniqueName: \"kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.198920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.198756 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.199668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.199642 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.202070 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.202047 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.202693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.202654 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7"] Oct 30 06:17:22.223734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.223697 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zhjv\" (UniqueName: \"kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv\") pod \"payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.299850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.299761 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kzv6\" (UniqueName: \"kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6\") pod \"payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.299850 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.299816 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.401144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.401107 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kzv6\" (UniqueName: \"kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6\") pod \"payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.401336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.401165 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.401898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.401873 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.452223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.450188 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kzv6\" (UniqueName: \"kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6\") pod \"payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.456889 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.456675 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:22.524245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.524193 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:22.804558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.803949 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk"] Oct 30 06:17:22.950077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:22.950051 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7"] Oct 30 06:17:23.592452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.592420 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd"] Oct 30 06:17:23.606247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.606220 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-698ab2-zs8fd"] Oct 30 06:17:23.830710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.830671 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" event={"ID":"589bbae7-449b-4aa2-9fa2-20a5b48d09b1","Type":"ContainerStarted","Data":"37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399"} Oct 30 06:17:23.830889 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.830735 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:23.830889 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.830757 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" event={"ID":"589bbae7-449b-4aa2-9fa2-20a5b48d09b1","Type":"ContainerStarted","Data":"909e7732b4026c23f6fc7e971182d71efb872744bed253773e7e769a0bc2c8ce"} Oct 30 06:17:23.832445 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.832407 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" event={"ID":"baaa1333-908d-4961-8812-675996545881","Type":"ContainerStarted","Data":"0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68"} Oct 30 06:17:23.832602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.832467 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" event={"ID":"baaa1333-908d-4961-8812-675996545881","Type":"ContainerStarted","Data":"153ec704aad21a42e48db7b85ac560601727b7597346549b337604738212fb68"} Oct 30 06:17:23.832602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.832501 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:23.886326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.886214 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" podStartSLOduration=2.886199601 podStartE2EDuration="2.886199601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:23.881123063 +0000 UTC m=+2750.456810120" watchObservedRunningTime="2026-10-30 06:17:23.886199601 +0000 UTC m=+2750.461886678" Oct 30 06:17:23.937154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:23.937106 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" podStartSLOduration=1.937092035 podStartE2EDuration="1.937092035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:23.934913014 +0000 UTC m=+2750.510600069" watchObservedRunningTime="2026-10-30 06:17:23.937092035 +0000 UTC m=+2750.512779125" Oct 30 06:17:25.569210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:25.569174 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7240fea0-9d52-4f59-9c3a-5ef205ebf95c" path="/var/lib/kubelet/pods/7240fea0-9d52-4f59-9c3a-5ef205ebf95c/volumes" Oct 30 06:17:30.475185 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:30.475147 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf"] Oct 30 06:17:30.475623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:30.475444 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" containerID="cri-o://89a0b1520331f9e34a35cb740fb9f3905f312fb5c7ed063fbd142115bf734fbe" gracePeriod=30 Oct 30 06:17:30.560054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:30.559915 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm"] Oct 30 06:17:34.843038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:34.843004 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:17:34.843658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:34.843636 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:17:35.088371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.088334 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp"] Oct 30 06:17:35.088696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.088652 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" podUID="517508c8-aeb3-425a-8ea4-3720a23aee80" containerName="maas-api" containerID="cri-o://6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5" gracePeriod=30 Oct 30 06:17:35.235355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.235313 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b"] Oct 30 06:17:35.235597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.235570 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" podUID="2753399b-3bdc-4c2a-b41b-c941c1af7271" containerName="payload-processing" containerID="cri-o://40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece" gracePeriod=30 Oct 30 06:17:35.451596 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.451570 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:17:35.562179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.562142 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" podUID="146e5605-cb4f-410e-ae48-b2e8fbdedd4a" containerName="payload-processing" containerID="cri-o://2d2442c92aaf02336e52859bb397a444591748ba2781da5a7b9c4220bbbb6b87" gracePeriod=25 Oct 30 06:17:35.599642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.599616 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs\") pod \"517508c8-aeb3-425a-8ea4-3720a23aee80\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " Oct 30 06:17:35.599770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.599661 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\") pod \"517508c8-aeb3-425a-8ea4-3720a23aee80\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " Oct 30 06:17:35.599840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.599816 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca\") pod \"517508c8-aeb3-425a-8ea4-3720a23aee80\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " Oct 30 06:17:35.599945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.599930 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f\" (UniqueName: \"kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f\") pod \"517508c8-aeb3-425a-8ea4-3720a23aee80\" (UID: \"517508c8-aeb3-425a-8ea4-3720a23aee80\") " Oct 30 06:17:35.600112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.600091 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca" pod "517508c8-aeb3-425a-8ea4-3720a23aee80" (UID: "517508c8-aeb3-425a-8ea4-3720a23aee80"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:35.600491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.600471 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/517508c8-aeb3-425a-8ea4-3720a23aee80-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.601659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.601628 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls" pod "517508c8-aeb3-425a-8ea4-3720a23aee80" (UID: "517508c8-aeb3-425a-8ea4-3720a23aee80"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:35.601761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.601677 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs" pod "517508c8-aeb3-425a-8ea4-3720a23aee80" (UID: "517508c8-aeb3-425a-8ea4-3720a23aee80"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:35.601862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.601840 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f" pod "517508c8-aeb3-425a-8ea4-3720a23aee80" (UID: "517508c8-aeb3-425a-8ea4-3720a23aee80"). InnerVolumeSpecName "kube-api-access-8lv7f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:35.710044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.709933 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lv7f\" (UniqueName: \"kubernetes.io/projected/517508c8-aeb3-425a-8ea4-3720a23aee80-kube-api-access-8lv7f\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.710044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.709969 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.710044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.709983 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/517508c8-aeb3-425a-8ea4-3720a23aee80-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.720852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.720821 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:17:35.810872 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.810833 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume\") pod \"2753399b-3bdc-4c2a-b41b-c941c1af7271\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " Oct 30 06:17:35.811939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.811910 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97\" (UniqueName: \"kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97\") pod \"2753399b-3bdc-4c2a-b41b-c941c1af7271\" (UID: \"2753399b-3bdc-4c2a-b41b-c941c1af7271\") " Oct 30 06:17:35.812627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.811491 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume" pod "2753399b-3bdc-4c2a-b41b-c941c1af7271" (UID: "2753399b-3bdc-4c2a-b41b-c941c1af7271"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:35.812998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.812974 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2753399b-3bdc-4c2a-b41b-c941c1af7271-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.823876 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.823825 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97" pod "2753399b-3bdc-4c2a-b41b-c941c1af7271" (UID: "2753399b-3bdc-4c2a-b41b-c941c1af7271"). InnerVolumeSpecName "kube-api-access-pdw97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:35.915124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.915087 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdw97\" (UniqueName: \"kubernetes.io/projected/2753399b-3bdc-4c2a-b41b-c941c1af7271-kube-api-access-pdw97\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:35.957838 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.957800 5565 generic.go:356] "Generic (PLEG): container finished" podID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerID="89a0b1520331f9e34a35cb740fb9f3905f312fb5c7ed063fbd142115bf734fbe" exitCode=0 Oct 30 06:17:35.958034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.957870 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" event={"ID":"6f573711-e9e1-4c7d-8ed4-b4b6976062e4","Type":"ContainerDied","Data":"89a0b1520331f9e34a35cb740fb9f3905f312fb5c7ed063fbd142115bf734fbe"} Oct 30 06:17:35.959791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.959766 5565 generic.go:356] "Generic (PLEG): container finished" podID="517508c8-aeb3-425a-8ea4-3720a23aee80" containerID="6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5" exitCode=0 Oct 30 06:17:35.959946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.959876 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" Oct 30 06:17:35.959946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.959880 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" event={"ID":"517508c8-aeb3-425a-8ea4-3720a23aee80","Type":"ContainerDied","Data":"6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5"} Oct 30 06:17:35.960079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.959931 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp" event={"ID":"517508c8-aeb3-425a-8ea4-3720a23aee80","Type":"ContainerDied","Data":"e15c774d17ac9ac6e724f4cce203447e786f9cba24b57757ad75ff5939fc115f"} Oct 30 06:17:35.960079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.959985 5565 scope.go:122] "RemoveContainer" containerID="6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5" Oct 30 06:17:35.966555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.962820 5565 generic.go:356] "Generic (PLEG): container finished" podID="2753399b-3bdc-4c2a-b41b-c941c1af7271" containerID="40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece" exitCode=0 Oct 30 06:17:35.966555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.962909 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" event={"ID":"2753399b-3bdc-4c2a-b41b-c941c1af7271","Type":"ContainerDied","Data":"40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece"} Oct 30 06:17:35.966555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.962950 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" event={"ID":"2753399b-3bdc-4c2a-b41b-c941c1af7271","Type":"ContainerDied","Data":"bf2f5277657e1cfd9b58f6407875bf2dcf8ba455b0f127510ce11493bcc9154e"} Oct 30 06:17:35.966555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.963031 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b" Oct 30 06:17:35.971345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.971313 5565 generic.go:356] "Generic (PLEG): container finished" podID="146e5605-cb4f-410e-ae48-b2e8fbdedd4a" containerID="2d2442c92aaf02336e52859bb397a444591748ba2781da5a7b9c4220bbbb6b87" exitCode=0 Oct 30 06:17:35.971498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.971364 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" event={"ID":"146e5605-cb4f-410e-ae48-b2e8fbdedd4a","Type":"ContainerDied","Data":"2d2442c92aaf02336e52859bb397a444591748ba2781da5a7b9c4220bbbb6b87"} Oct 30 06:17:35.980575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.980237 5565 scope.go:122] "RemoveContainer" containerID="6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5" Oct 30 06:17:35.980906 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:17:35.980741 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5\": container with ID starting with 6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5 not found: ID does not exist" containerID="6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5" Oct 30 06:17:35.980906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.980786 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5"} err="failed to get container status \"6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5\": rpc error: code = NotFound desc = could not find container \"6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5\": container with ID starting with 6e70de5679fb87654be0218b34832d81a71cefc1358a81295b4ffca88e7789a5 not found: ID does not exist" Oct 30 06:17:35.980906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.980811 5565 scope.go:122] "RemoveContainer" containerID="40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece" Oct 30 06:17:35.998171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.997922 5565 scope.go:122] "RemoveContainer" containerID="40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece" Oct 30 06:17:35.998313 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:17:35.998290 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece\": container with ID starting with 40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece not found: ID does not exist" containerID="40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece" Oct 30 06:17:35.998392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:35.998326 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece"} err="failed to get container status \"40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece\": rpc error: code = NotFound desc = could not find container \"40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece\": container with ID starting with 40ddaefdfebd6fd1d57709ccbdfdd5006b5b97a198d2896006ef6398c28f3ece not found: ID does not exist" Oct 30 06:17:36.005503 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.005477 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 06:17:36.016792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.016751 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.016863 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.016905 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.016948 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.017059 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.017122 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.017193 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.017250 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.017431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.017290 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd\") pod \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\" (UID: \"6f573711-e9e1-4c7d-8ed4-b4b6976062e4\") " Oct 30 06:17:36.019478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.019432 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:36.019843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.019811 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:36.021325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.020875 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:36.021325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021081 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:36.021325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021297 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:36.021646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021582 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.021646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021641 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.021783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021656 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.021783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021670 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.021783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.021712 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.022982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.022469 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:17:36.025187 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.025101 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:17:36.025187 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.025139 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:36.027477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.027378 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x"] Oct 30 06:17:36.027477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.027408 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd" pod "6f573711-e9e1-4c7d-8ed4-b4b6976062e4" (UID: "6f573711-e9e1-4c7d-8ed4-b4b6976062e4"). InnerVolumeSpecName "kube-api-access-gz7kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:36.034076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.034042 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp"] Oct 30 06:17:36.038633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.038602 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:17:36.039713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.039692 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-c201cd-pl52x"] Oct 30 06:17:36.044644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.044615 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-c201cd-86656bfc68-6qjzp"] Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.122736 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume\") pod \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.122934 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv\" (UniqueName: \"kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv\") pod \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\" (UID: \"146e5605-cb4f-410e-ae48-b2e8fbdedd4a\") " Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.123727 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.123750 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.123767 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.123782 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gz7kd\" (UniqueName: \"kubernetes.io/projected/6f573711-e9e1-4c7d-8ed4-b4b6976062e4-kube-api-access-gz7kd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.127968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.124011 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume" pod "146e5605-cb4f-410e-ae48-b2e8fbdedd4a" (UID: "146e5605-cb4f-410e-ae48-b2e8fbdedd4a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:36.133001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.132957 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv" pod "146e5605-cb4f-410e-ae48-b2e8fbdedd4a" (UID: "146e5605-cb4f-410e-ae48-b2e8fbdedd4a"). InnerVolumeSpecName "kube-api-access-6gswv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:36.173703 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.173670 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b"] Oct 30 06:17:36.179750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.179720 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-c201cd-857757fdfb-stb5b"] Oct 30 06:17:36.225750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.225617 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6gswv\" (UniqueName: \"kubernetes.io/projected/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-kube-api-access-6gswv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.225750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.225650 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/146e5605-cb4f-410e-ae48-b2e8fbdedd4a-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:36.981617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.981581 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" Oct 30 06:17:36.982135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.981588 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf" event={"ID":"6f573711-e9e1-4c7d-8ed4-b4b6976062e4","Type":"ContainerDied","Data":"061f1897118f37b1ed93a3a2435e99bad86d251ec9ef1c532b7b8bdf5b37730e"} Oct 30 06:17:36.982135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.981724 5565 scope.go:122] "RemoveContainer" containerID="89a0b1520331f9e34a35cb740fb9f3905f312fb5c7ed063fbd142115bf734fbe" Oct 30 06:17:36.986076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.986047 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" Oct 30 06:17:36.986230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.986051 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm" event={"ID":"146e5605-cb4f-410e-ae48-b2e8fbdedd4a","Type":"ContainerDied","Data":"225e9d48d668e2580406269844f248ad2246a321f34ed4e2a7b66bf016dcd9b3"} Oct 30 06:17:36.994630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:36.994610 5565 scope.go:122] "RemoveContainer" containerID="2d2442c92aaf02336e52859bb397a444591748ba2781da5a7b9c4220bbbb6b87" Oct 30 06:17:37.041277 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.041240 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm"] Oct 30 06:17:37.046814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.046780 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-5c7458cd9d-b5vnm"] Oct 30 06:17:37.085168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.085124 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf"] Oct 30 06:17:37.094000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.093961 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-698ab2-openshift-default-646d4c697-85rxf"] Oct 30 06:17:37.570891 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.570825 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="146e5605-cb4f-410e-ae48-b2e8fbdedd4a" path="/var/lib/kubelet/pods/146e5605-cb4f-410e-ae48-b2e8fbdedd4a/volumes" Oct 30 06:17:37.571458 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.571434 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2753399b-3bdc-4c2a-b41b-c941c1af7271" path="/var/lib/kubelet/pods/2753399b-3bdc-4c2a-b41b-c941c1af7271/volumes" Oct 30 06:17:37.572025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.572003 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="517508c8-aeb3-425a-8ea4-3720a23aee80" path="/var/lib/kubelet/pods/517508c8-aeb3-425a-8ea4-3720a23aee80/volumes" Oct 30 06:17:37.573119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.572849 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5672b652-40ea-41dd-ba82-6bcc491caea5" path="/var/lib/kubelet/pods/5672b652-40ea-41dd-ba82-6bcc491caea5/volumes" Oct 30 06:17:37.573671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.573486 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" path="/var/lib/kubelet/pods/6f573711-e9e1-4c7d-8ed4-b4b6976062e4/volumes" Oct 30 06:17:37.863643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.854383 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv"] Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870677 5565 cpu_manager.go:414] "containerMap: removing container" podUID="517508c8-aeb3-425a-8ea4-3720a23aee80" containerName="maas-api" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870715 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="517508c8-aeb3-425a-8ea4-3720a23aee80" containerName="maas-api" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870762 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2753399b-3bdc-4c2a-b41b-c941c1af7271" containerName="payload-processing" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870781 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2753399b-3bdc-4c2a-b41b-c941c1af7271" containerName="payload-processing" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870842 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870850 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870879 5565 cpu_manager.go:414] "containerMap: removing container" podUID="146e5605-cb4f-410e-ae48-b2e8fbdedd4a" containerName="payload-processing" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.870887 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="146e5605-cb4f-410e-ae48-b2e8fbdedd4a" containerName="payload-processing" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.871198 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f573711-e9e1-4c7d-8ed4-b4b6976062e4" containerName="istio-proxy" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.871220 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="146e5605-cb4f-410e-ae48-b2e8fbdedd4a" containerName="payload-processing" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.871239 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="517508c8-aeb3-425a-8ea4-3720a23aee80" containerName="maas-api" Oct 30 06:17:37.877552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.871256 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2753399b-3bdc-4c2a-b41b-c941c1af7271" containerName="payload-processing" Oct 30 06:17:37.909565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.903593 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv"] Oct 30 06:17:37.909565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.903756 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:37.947158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.947120 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-de2bf2-dockercfg-m8ztm\"" Oct 30 06:17:37.947640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.947613 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-de2bf2\"" Oct 30 06:17:37.999171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:37.999139 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk"] Oct 30 06:17:38.005314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.005276 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.024434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.024401 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk"] Oct 30 06:17:38.057967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.057916 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg58l\" (UniqueName: \"kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l\") pod \"payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.058317 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.058286 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2m498\" (UniqueName: \"kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.058404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.058388 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.058441 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.058426 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.159954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.159854 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2m498\" (UniqueName: \"kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.159954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.159907 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.159954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.159944 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.160658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.160045 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg58l\" (UniqueName: \"kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l\") pod \"payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.160833 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.160812 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.160909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.160892 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.188050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.188013 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2m498\" (UniqueName: \"kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498\") pod \"payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.188199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.188081 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg58l\" (UniqueName: \"kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l\") pod \"payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.228472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.228428 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:38.325164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.325119 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:38.571622 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.553858 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv"] Oct 30 06:17:38.571622 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:17:38.555997 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70ea4097_fdd9_4811_aa82_1b5d8423de58.slice/crio-53cfc4706b5849d118852b61499464a11d8d5ddcaa7920eda8bf9b73da164ae6 WatchSource:0}: Error finding container 53cfc4706b5849d118852b61499464a11d8d5ddcaa7920eda8bf9b73da164ae6: Status 404 returned error can't find the container with id 53cfc4706b5849d118852b61499464a11d8d5ddcaa7920eda8bf9b73da164ae6 Oct 30 06:17:38.660221 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:17:38.660192 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fbb7b13_671d_4383_bef4_2cf446b2e99b.slice/crio-494b8f715de73c3dbdaafc0c264e401e3f91a06f67d45d0cb7fd3ca4822cdf7b WatchSource:0}: Error finding container 494b8f715de73c3dbdaafc0c264e401e3f91a06f67d45d0cb7fd3ca4822cdf7b: Status 404 returned error can't find the container with id 494b8f715de73c3dbdaafc0c264e401e3f91a06f67d45d0cb7fd3ca4822cdf7b Oct 30 06:17:38.661224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.661197 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk"] Oct 30 06:17:38.858654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.858621 5565 scope.go:122] "RemoveContainer" containerID="2982141c01f810595dae3ead09af58689de516e0f38de0e511ebbc06f68d7efe" Oct 30 06:17:38.872442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.872415 5565 scope.go:122] "RemoveContainer" containerID="f2e17eb72f238db4c0d93a9b53f88f5a84c1cf3d488463bd58a7653eea41b870" Oct 30 06:17:38.884643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.884617 5565 scope.go:122] "RemoveContainer" containerID="f9abfaabc61c77793bc982da96d25587ae8ff0fd79567399f3311d34531b38d4" Oct 30 06:17:38.900045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.899869 5565 scope.go:122] "RemoveContainer" containerID="aca1205b8518e597178798885ed218082f837251b69104dd5dd76db0cb8b043c" Oct 30 06:17:38.914363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:38.914337 5565 scope.go:122] "RemoveContainer" containerID="cbc0defa9827664b7a8030edddbbd53e892a0d3bb3292f5900aac1c85902f0d8" Oct 30 06:17:39.015054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.015006 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" event={"ID":"70ea4097-fdd9-4811-aa82-1b5d8423de58","Type":"ContainerStarted","Data":"0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91"} Oct 30 06:17:39.015506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.015074 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:39.015506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.015100 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" event={"ID":"70ea4097-fdd9-4811-aa82-1b5d8423de58","Type":"ContainerStarted","Data":"53cfc4706b5849d118852b61499464a11d8d5ddcaa7920eda8bf9b73da164ae6"} Oct 30 06:17:39.017118 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.017088 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" event={"ID":"4fbb7b13-671d-4383-bef4-2cf446b2e99b","Type":"ContainerStarted","Data":"5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab"} Oct 30 06:17:39.017244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.017136 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" event={"ID":"4fbb7b13-671d-4383-bef4-2cf446b2e99b","Type":"ContainerStarted","Data":"494b8f715de73c3dbdaafc0c264e401e3f91a06f67d45d0cb7fd3ca4822cdf7b"} Oct 30 06:17:39.017244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.017162 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:39.062240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.062183 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" podStartSLOduration=2.062166122 podStartE2EDuration="2.062166122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:39.060196194 +0000 UTC m=+2765.635883250" watchObservedRunningTime="2026-10-30 06:17:39.062166122 +0000 UTC m=+2765.637853175" Oct 30 06:17:39.108574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:39.108496 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" podStartSLOduration=2.108480724 podStartE2EDuration="2.108480724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:17:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:17:39.104041457 +0000 UTC m=+2765.679728536" watchObservedRunningTime="2026-10-30 06:17:39.108480724 +0000 UTC m=+2765.684167824" Oct 30 06:17:43.516267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.516226 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf"] Oct 30 06:17:43.523063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.523039 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.545010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.544980 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf"] Oct 30 06:17:43.623140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.623112 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:17:43.623278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.623139 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:17:43.693296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.693270 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.693424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.693324 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.693589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.693566 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7p2l\" (UniqueName: \"kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.794811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.794750 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7p2l\" (UniqueName: \"kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.794944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.794928 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.794986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.794964 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.795601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.795578 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.797217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.797198 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.822955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.822934 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7p2l\" (UniqueName: \"kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l\") pod \"maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:43.838913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:43.838890 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:44.156122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:44.156086 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf"] Oct 30 06:17:44.156316 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:17:44.156294 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4faaa2e_e6cd_4685_b6e2_e721525be90d.slice/crio-1729d06b19f6ed1b7591c07585103804a8932427cdcf0767a9074a1e77219d30 WatchSource:0}: Error finding container 1729d06b19f6ed1b7591c07585103804a8932427cdcf0767a9074a1e77219d30: Status 404 returned error can't find the container with id 1729d06b19f6ed1b7591c07585103804a8932427cdcf0767a9074a1e77219d30 Oct 30 06:17:45.108920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:45.108891 5565 generic.go:356] "Generic (PLEG): container finished" podID="d4faaa2e-e6cd-4685-b6e2-e721525be90d" containerID="71e999277be4ee6f165e05d084d93848c2c36d10a02980a8119515ed97bb9c33" exitCode=0 Oct 30 06:17:45.109310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:45.108979 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" event={"ID":"d4faaa2e-e6cd-4685-b6e2-e721525be90d","Type":"ContainerDied","Data":"71e999277be4ee6f165e05d084d93848c2c36d10a02980a8119515ed97bb9c33"} Oct 30 06:17:45.109310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:45.109046 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" event={"ID":"d4faaa2e-e6cd-4685-b6e2-e721525be90d","Type":"ContainerStarted","Data":"1729d06b19f6ed1b7591c07585103804a8932427cdcf0767a9074a1e77219d30"} Oct 30 06:17:46.327490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.327465 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:46.363149 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.363124 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l\" (UniqueName: \"kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l\") pod \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " Oct 30 06:17:46.363286 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.363213 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca\") pod \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " Oct 30 06:17:46.363342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.363330 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca\") pod \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\" (UID: \"d4faaa2e-e6cd-4685-b6e2-e721525be90d\") " Oct 30 06:17:46.363706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.363677 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca" pod "d4faaa2e-e6cd-4685-b6e2-e721525be90d" (UID: "d4faaa2e-e6cd-4685-b6e2-e721525be90d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:46.363853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.363828 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:46.365294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.365264 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca" pod "d4faaa2e-e6cd-4685-b6e2-e721525be90d" (UID: "d4faaa2e-e6cd-4685-b6e2-e721525be90d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:46.365427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.365303 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l" pod "d4faaa2e-e6cd-4685-b6e2-e721525be90d" (UID: "d4faaa2e-e6cd-4685-b6e2-e721525be90d"). InnerVolumeSpecName "kube-api-access-s7p2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:46.486552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.482485 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7p2l\" (UniqueName: \"kubernetes.io/projected/d4faaa2e-e6cd-4685-b6e2-e721525be90d-kube-api-access-s7p2l\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:46.486552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:46.482518 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d4faaa2e-e6cd-4685-b6e2-e721525be90d-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:47.124168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:47.124140 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" Oct 30 06:17:47.124356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:47.124177 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf" event={"ID":"d4faaa2e-e6cd-4685-b6e2-e721525be90d","Type":"ContainerDied","Data":"1729d06b19f6ed1b7591c07585103804a8932427cdcf0767a9074a1e77219d30"} Oct 30 06:17:47.124356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:47.124229 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1729d06b19f6ed1b7591c07585103804a8932427cdcf0767a9074a1e77219d30" Oct 30 06:17:47.534824 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:17:47.534783 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:17:50.026732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:50.026700 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:17:50.027146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:50.026776 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:17:56.197090 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.197055 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg"] Oct 30 06:17:56.198184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.198163 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d4faaa2e-e6cd-4685-b6e2-e721525be90d" containerName="revoke-keys" Oct 30 06:17:56.198184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.198183 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4faaa2e-e6cd-4685-b6e2-e721525be90d" containerName="revoke-keys" Oct 30 06:17:56.198364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.198352 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4faaa2e-e6cd-4685-b6e2-e721525be90d" containerName="revoke-keys" Oct 30 06:17:56.202289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.202269 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.222319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.222295 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg"] Oct 30 06:17:56.235761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.235739 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:17:56.235885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.235850 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:17:56.353136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.353097 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.353305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.353188 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rlc7\" (UniqueName: \"kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.353305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.353269 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.454930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.454843 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.454930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.454892 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rlc7\" (UniqueName: \"kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.455171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.454941 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.455617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.455596 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.457643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.457620 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.480111 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.480086 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rlc7\" (UniqueName: \"kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7\") pod \"maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.522327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.522303 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:56.718423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.718339 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm"] Oct 30 06:17:56.718609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.718546 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" podUID="0aee4521-1565-4fd5-9e1f-e41227ad2ae7" containerName="payload-processing" containerID="cri-o://e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed" gracePeriod=30 Oct 30 06:17:56.733785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.733756 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24"] Oct 30 06:17:56.734068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.734039 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" podUID="a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" containerName="payload-pre-processing" containerID="cri-o://85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6" gracePeriod=30 Oct 30 06:17:56.837663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:56.837636 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg"] Oct 30 06:17:56.867133 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:17:56.867105 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod302ab598_206a_41a5_a026_12229cf1469b.slice/crio-778b49080a88356e60848dee2a1919f2037f1e87865a1afd99513e3e599d265c WatchSource:0}: Error finding container 778b49080a88356e60848dee2a1919f2037f1e87865a1afd99513e3e599d265c: Status 404 returned error can't find the container with id 778b49080a88356e60848dee2a1919f2037f1e87865a1afd99513e3e599d265c Oct 30 06:17:57.013354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.013325 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:17:57.070492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.070459 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:17:57.173571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.172083 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq\" (UniqueName: \"kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq\") pod \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " Oct 30 06:17:57.173571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.172161 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume\") pod \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " Oct 30 06:17:57.173571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.172333 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume\") pod \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\" (UID: \"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb\") " Oct 30 06:17:57.173571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.172674 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc\" (UniqueName: \"kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc\") pod \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\" (UID: \"0aee4521-1565-4fd5-9e1f-e41227ad2ae7\") " Oct 30 06:17:57.186586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.173999 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume" pod "0aee4521-1565-4fd5-9e1f-e41227ad2ae7" (UID: "0aee4521-1565-4fd5-9e1f-e41227ad2ae7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:57.280132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.278189 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.334875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.333489 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq" pod "a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" (UID: "a44482ec-6a3c-4c20-9d35-83b0e1aea0bb"). InnerVolumeSpecName "kube-api-access-5bgbq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:57.344083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.344048 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc" pod "0aee4521-1565-4fd5-9e1f-e41227ad2ae7" (UID: "0aee4521-1565-4fd5-9e1f-e41227ad2ae7"). InnerVolumeSpecName "kube-api-access-7qqkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:57.355195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.355158 5565 generic.go:356] "Generic (PLEG): container finished" podID="a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" containerID="85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6" exitCode=0 Oct 30 06:17:57.355339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.355282 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" Oct 30 06:17:57.355339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.355298 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" event={"ID":"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb","Type":"ContainerDied","Data":"85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6"} Oct 30 06:17:57.355419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.355349 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24" event={"ID":"a44482ec-6a3c-4c20-9d35-83b0e1aea0bb","Type":"ContainerDied","Data":"526ddcb7e8266fba81021aa3631dd28e8e45da99236be7ffff4d6502f0b3df05"} Oct 30 06:17:57.355419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.355379 5565 scope.go:122] "RemoveContainer" containerID="85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6" Oct 30 06:17:57.356780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.356749 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume" pod "a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" (UID: "a44482ec-6a3c-4c20-9d35-83b0e1aea0bb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:57.358037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.357739 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" event={"ID":"302ab598-206a-41a5-a026-12229cf1469b","Type":"ContainerStarted","Data":"778b49080a88356e60848dee2a1919f2037f1e87865a1afd99513e3e599d265c"} Oct 30 06:17:57.360077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.360054 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" Oct 30 06:17:57.363066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.363039 5565 generic.go:356] "Generic (PLEG): container finished" podID="0aee4521-1565-4fd5-9e1f-e41227ad2ae7" containerID="e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed" exitCode=0 Oct 30 06:17:57.363445 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.363192 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" event={"ID":"0aee4521-1565-4fd5-9e1f-e41227ad2ae7","Type":"ContainerDied","Data":"e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed"} Oct 30 06:17:57.363657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.363639 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm" event={"ID":"0aee4521-1565-4fd5-9e1f-e41227ad2ae7","Type":"ContainerDied","Data":"88ae92b13eab135eb2f1bc65e2017a66664e0fb75c53ae7a6d884ee816527dc5"} Oct 30 06:17:57.374696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.374672 5565 scope.go:122] "RemoveContainer" containerID="85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6" Oct 30 06:17:57.374953 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:17:57.374934 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6\": container with ID starting with 85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6 not found: ID does not exist" containerID="85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6" Oct 30 06:17:57.375016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.374959 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6"} err="failed to get container status \"85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6\": rpc error: code = NotFound desc = could not find container \"85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6\": container with ID starting with 85a744f2a3177bb3334ab533f9c8e6ea0176d81d4b9c60a491e058c623cfa7a6 not found: ID does not exist" Oct 30 06:17:57.375016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.374975 5565 scope.go:122] "RemoveContainer" containerID="e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed" Oct 30 06:17:57.380185 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.380159 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.380271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.380192 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qqkc\" (UniqueName: \"kubernetes.io/projected/0aee4521-1565-4fd5-9e1f-e41227ad2ae7-kube-api-access-7qqkc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.380271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.380209 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bgbq\" (UniqueName: \"kubernetes.io/projected/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb-kube-api-access-5bgbq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:57.385826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.385807 5565 scope.go:122] "RemoveContainer" containerID="e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed" Oct 30 06:17:57.386082 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:17:57.386062 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed\": container with ID starting with e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed not found: ID does not exist" containerID="e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed" Oct 30 06:17:57.386144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.386090 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed"} err="failed to get container status \"e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed\": rpc error: code = NotFound desc = could not find container \"e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed\": container with ID starting with e339aa42ca943e6e43765de9e791973aacc82e9bc1a0a1917300defd891097ed not found: ID does not exist" Oct 30 06:17:57.418955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.418927 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm"] Oct 30 06:17:57.422729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.422708 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-370dfa-6799dc4bb6-nttqm"] Oct 30 06:17:57.445010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.444985 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf"] Oct 30 06:17:57.448263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.448244 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-370dfa-gqmdf"] Oct 30 06:17:57.569294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.569221 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aee4521-1565-4fd5-9e1f-e41227ad2ae7" path="/var/lib/kubelet/pods/0aee4521-1565-4fd5-9e1f-e41227ad2ae7/volumes" Oct 30 06:17:57.569779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.569763 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4faaa2e-e6cd-4685-b6e2-e721525be90d" path="/var/lib/kubelet/pods/d4faaa2e-e6cd-4685-b6e2-e721525be90d/volumes" Oct 30 06:17:57.701773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.701738 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24"] Oct 30 06:17:57.706365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:57.706333 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-370dfa-864699bff9-77n24"] Oct 30 06:17:58.374882 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:58.374794 5565 generic.go:356] "Generic (PLEG): container finished" podID="302ab598-206a-41a5-a026-12229cf1469b" containerID="9206785fefaef55656483806518b961fc4c9863e9e840a11a6492773fa827d41" exitCode=0 Oct 30 06:17:58.375271 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:58.374872 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" event={"ID":"302ab598-206a-41a5-a026-12229cf1469b","Type":"ContainerDied","Data":"9206785fefaef55656483806518b961fc4c9863e9e840a11a6492773fa827d41"} Oct 30 06:17:59.561738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.561706 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:17:59.571422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.571392 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" path="/var/lib/kubelet/pods/a44482ec-6a3c-4c20-9d35-83b0e1aea0bb/volumes" Oct 30 06:17:59.645285 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.645208 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7\" (UniqueName: \"kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7\") pod \"302ab598-206a-41a5-a026-12229cf1469b\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " Oct 30 06:17:59.645446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.645404 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca\") pod \"302ab598-206a-41a5-a026-12229cf1469b\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " Oct 30 06:17:59.645541 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.645500 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca\") pod \"302ab598-206a-41a5-a026-12229cf1469b\" (UID: \"302ab598-206a-41a5-a026-12229cf1469b\") " Oct 30 06:17:59.646018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.645991 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca" pod "302ab598-206a-41a5-a026-12229cf1469b" (UID: "302ab598-206a-41a5-a026-12229cf1469b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:17:59.647635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.647545 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/302ab598-206a-41a5-a026-12229cf1469b-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:59.648643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.648622 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca" pod "302ab598-206a-41a5-a026-12229cf1469b" (UID: "302ab598-206a-41a5-a026-12229cf1469b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:17:59.658864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.658833 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7" pod "302ab598-206a-41a5-a026-12229cf1469b" (UID: "302ab598-206a-41a5-a026-12229cf1469b"). InnerVolumeSpecName "kube-api-access-2rlc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:17:59.749072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.749031 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rlc7\" (UniqueName: \"kubernetes.io/projected/302ab598-206a-41a5-a026-12229cf1469b-kube-api-access-2rlc7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:17:59.749072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:17:59.749066 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/302ab598-206a-41a5-a026-12229cf1469b-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:00.397080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:00.397049 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" Oct 30 06:18:00.397266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:00.397091 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg" event={"ID":"302ab598-206a-41a5-a026-12229cf1469b","Type":"ContainerDied","Data":"778b49080a88356e60848dee2a1919f2037f1e87865a1afd99513e3e599d265c"} Oct 30 06:18:00.397266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:00.397146 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="778b49080a88356e60848dee2a1919f2037f1e87865a1afd99513e3e599d265c" Oct 30 06:18:04.406309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.406267 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg"] Oct 30 06:18:04.407863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.407795 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" containerID="cri-o://4e74a78ce719d1c75aa00fd323644ca349319f1bc8c4f0e6cdd0a492d23b9c0d" gracePeriod=30 Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.952920 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x"] Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954077 5565 cpu_manager.go:414] "containerMap: removing container" podUID="302ab598-206a-41a5-a026-12229cf1469b" containerName="revoke-keys" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954100 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="302ab598-206a-41a5-a026-12229cf1469b" containerName="revoke-keys" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954160 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0aee4521-1565-4fd5-9e1f-e41227ad2ae7" containerName="payload-processing" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954169 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aee4521-1565-4fd5-9e1f-e41227ad2ae7" containerName="payload-processing" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954206 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" containerName="payload-pre-processing" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954214 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" containerName="payload-pre-processing" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954385 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="302ab598-206a-41a5-a026-12229cf1469b" containerName="revoke-keys" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954406 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aee4521-1565-4fd5-9e1f-e41227ad2ae7" containerName="payload-processing" Oct 30 06:18:04.961553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.954421 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a44482ec-6a3c-4c20-9d35-83b0e1aea0bb" containerName="payload-pre-processing" Oct 30 06:18:04.985234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.985207 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:04.994773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:04.991134 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x"] Oct 30 06:18:05.002563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.002484 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:18:05.002748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.002707 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:18:05.096928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.096889 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.097104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.097077 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7qc\" (UniqueName: \"kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.097174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.097148 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.201597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.201310 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.201597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.201572 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.201883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.201757 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7qc\" (UniqueName: \"kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.206563 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.206364 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.218716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.217655 5565 patch_prober.go:29] interesting pod/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:18:05.218716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.217764 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:18:05.219941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.219907 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.229928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.228712 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7qc\" (UniqueName: \"kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.303316 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.303274 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:05.650305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:05.650277 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x"] Oct 30 06:18:05.651229 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:18:05.651196 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2e4f4f_dd7e_434a_91aa_ac5f14727a1b.slice/crio-979846d013e783c76ddcb90c0b3125bbbb5af8083ede9dd31507bd6d188e1534 WatchSource:0}: Error finding container 979846d013e783c76ddcb90c0b3125bbbb5af8083ede9dd31507bd6d188e1534: Status 404 returned error can't find the container with id 979846d013e783c76ddcb90c0b3125bbbb5af8083ede9dd31507bd6d188e1534 Oct 30 06:18:06.500754 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:06.500657 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" event={"ID":"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b","Type":"ContainerStarted","Data":"3b148ffb0ff11e454a59bb28244c43e5422e23402870bf96a509ac1e2cb7a8b0"} Oct 30 06:18:06.500754 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:06.500728 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" event={"ID":"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b","Type":"ContainerStarted","Data":"979846d013e783c76ddcb90c0b3125bbbb5af8083ede9dd31507bd6d188e1534"} Oct 30 06:18:06.545982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:06.545928 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" podStartSLOduration=2.545914318 podStartE2EDuration="2.545914318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:06.544715079 +0000 UTC m=+2793.120402132" watchObservedRunningTime="2026-10-30 06:18:06.545914318 +0000 UTC m=+2793.121601384" Oct 30 06:18:07.512793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:07.512754 5565 generic.go:356] "Generic (PLEG): container finished" podID="7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" containerID="3b148ffb0ff11e454a59bb28244c43e5422e23402870bf96a509ac1e2cb7a8b0" exitCode=0 Oct 30 06:18:07.513219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:07.512840 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" event={"ID":"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b","Type":"ContainerDied","Data":"3b148ffb0ff11e454a59bb28244c43e5422e23402870bf96a509ac1e2cb7a8b0"} Oct 30 06:18:08.773996 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.773928 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:08.857687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.857653 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca\") pod \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " Oct 30 06:18:08.857890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.857768 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca\") pod \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " Oct 30 06:18:08.857890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.857845 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc\" (UniqueName: \"kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc\") pod \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\" (UID: \"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b\") " Oct 30 06:18:08.858281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.858186 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca" pod "7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" (UID: "7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:08.860194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.860166 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca" pod "7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" (UID: "7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:08.860513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.860456 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc" pod "7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" (UID: "7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b"). InnerVolumeSpecName "kube-api-access-kc7qc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:08.959544 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.959501 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:08.959544 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.959545 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:08.959749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:08.959555 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc7qc\" (UniqueName: \"kubernetes.io/projected/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b-kube-api-access-kc7qc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.532462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.532432 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" Oct 30 06:18:09.532623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.532458 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x" event={"ID":"7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b","Type":"ContainerDied","Data":"979846d013e783c76ddcb90c0b3125bbbb5af8083ede9dd31507bd6d188e1534"} Oct 30 06:18:09.532623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.532521 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="979846d013e783c76ddcb90c0b3125bbbb5af8083ede9dd31507bd6d188e1534" Oct 30 06:18:09.534608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.534585 5565 generic.go:356] "Generic (PLEG): container finished" podID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerID="4e74a78ce719d1c75aa00fd323644ca349319f1bc8c4f0e6cdd0a492d23b9c0d" exitCode=0 Oct 30 06:18:09.534723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.534653 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" event={"ID":"d078081e-660d-4c3a-8c14-dd61d34f9e82","Type":"ContainerDied","Data":"4e74a78ce719d1c75aa00fd323644ca349319f1bc8c4f0e6cdd0a492d23b9c0d"} Oct 30 06:18:09.534723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.534709 5565 scope.go:122] "RemoveContainer" containerID="46a0d14d45a7a0a93bbc4ef99a99f38b0be144615d59506eed0d9d2b38504ede" Oct 30 06:18:09.726170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.726141 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:18:09.769997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.769960 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770064 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770161 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770236 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770286 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770317 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770353 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:09.770438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770381 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770358 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:09.770438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770410 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770444 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo\") pod \"d078081e-660d-4c3a-8c14-dd61d34f9e82\" (UID: \"d078081e-660d-4c3a-8c14-dd61d34f9e82\") " Oct 30 06:18:09.770802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.770747 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:09.771076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.771050 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.771159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.771081 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d078081e-660d-4c3a-8c14-dd61d34f9e82-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.771159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.771096 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.771159 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.771089 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:09.771336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.771144 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:09.772678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.772653 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:09.772853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.772831 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:18:09.773015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.772998 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:18:09.773393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.773367 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b" pod "d078081e-660d-4c3a-8c14-dd61d34f9e82" (UID: "d078081e-660d-4c3a-8c14-dd61d34f9e82"). InnerVolumeSpecName "kube-api-access-2cb8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:09.872437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.872405 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.872437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.872432 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.872437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.872443 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2cb8b\" (UniqueName: \"kubernetes.io/projected/d078081e-660d-4c3a-8c14-dd61d34f9e82-kube-api-access-2cb8b\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.872879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.872453 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.872879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.872463 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d078081e-660d-4c3a-8c14-dd61d34f9e82-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:09.872879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:09.872471 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d078081e-660d-4c3a-8c14-dd61d34f9e82-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:10.545020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:10.544987 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" Oct 30 06:18:10.545221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:10.545020 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg" event={"ID":"d078081e-660d-4c3a-8c14-dd61d34f9e82","Type":"ContainerDied","Data":"95ac1663df6d7dd4ac185b68d113a2ed9cd2adc81a70af47c7a3c660b9fe23d0"} Oct 30 06:18:10.545221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:10.545088 5565 scope.go:122] "RemoveContainer" containerID="4e74a78ce719d1c75aa00fd323644ca349319f1bc8c4f0e6cdd0a492d23b9c0d" Oct 30 06:18:10.610358 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:10.610328 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg"] Oct 30 06:18:10.623719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:10.623695 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-370dfa-openshift-default-6f9d79469f-zhjdg"] Oct 30 06:18:11.227913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.227861 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:18:11.228251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.227909 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 06:18:11.554381 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.554349 5565 generic.go:356] "Generic (PLEG): container finished" podID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerID="073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403" exitCode=0 Oct 30 06:18:11.554564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.554420 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" event={"ID":"121ba31d-7c52-4d21-bac3-fc3848d2a8f4","Type":"ContainerDied","Data":"073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403"} Oct 30 06:18:11.554845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.554827 5565 scope.go:122] "RemoveContainer" containerID="073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403" Oct 30 06:18:11.556637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.556606 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:18:11.556724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.556680 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:18:11.556724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.556716 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:18:11.569749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:11.569726 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" path="/var/lib/kubelet/pods/d078081e-660d-4c3a-8c14-dd61d34f9e82/volumes" Oct 30 06:18:12.613241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:12.613208 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" event={"ID":"121ba31d-7c52-4d21-bac3-fc3848d2a8f4","Type":"ContainerStarted","Data":"6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9"} Oct 30 06:18:12.997987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:12.997898 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:18:12.998179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:12.998134 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:18:12.998263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:12.998193 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 06:18:14.003069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:14.003028 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:18:14.003959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:14.003921 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 06:18:14.998107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:14.998076 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:18:14.998325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:14.998147 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 06:18:15.340905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:15.340864 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4"] Oct 30 06:18:15.998196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:15.998157 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:18:15.998399 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:15.998210 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 06:18:16.292591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.275857 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s"] Oct 30 06:18:16.292591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.277142 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" containerName="revoke-keys" Oct 30 06:18:16.292591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.277165 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" containerName="revoke-keys" Oct 30 06:18:16.299076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.298967 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" Oct 30 06:18:16.299076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.298995 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" Oct 30 06:18:16.301934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.299585 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" containerName="revoke-keys" Oct 30 06:18:16.301934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.299617 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" Oct 30 06:18:16.301934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.299633 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" Oct 30 06:18:16.306720 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.306699 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.340140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.340099 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s"] Oct 30 06:18:16.347543 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.347503 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-fb9345-dockercfg-h6ks4\"" Oct 30 06:18:16.349671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.348620 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-fb9345\"" Oct 30 06:18:16.494435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.494403 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl6rz\" (UniqueName: \"kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz\") pod \"payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.494645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.494620 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.596686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.596596 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl6rz\" (UniqueName: \"kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz\") pod \"payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.596825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.596739 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.597479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.597458 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.624170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.624148 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl6rz\" (UniqueName: \"kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz\") pod \"payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.669551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.665766 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:16.998071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.997998 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" start-of-body= Oct 30 06:18:16.998071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:16.998051 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.13:15021/healthz/ready\": dial tcp 10.134.1.13:15021: connect: connection refused" Oct 30 06:18:17.037047 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:18:17.037018 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef84b6f_f535_4625_a871_4bc7597e373b.slice/crio-005c31c5f417fc9820059927590d824ee59a44655f3344a8d0b39c541f4ec50a WatchSource:0}: Error finding container 005c31c5f417fc9820059927590d824ee59a44655f3344a8d0b39c541f4ec50a: Status 404 returned error can't find the container with id 005c31c5f417fc9820059927590d824ee59a44655f3344a8d0b39c541f4ec50a Oct 30 06:18:17.037640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:17.037615 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s"] Oct 30 06:18:17.536980 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:18:17.536945 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:18:17.693167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:17.693129 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" event={"ID":"4ef84b6f-f535-4625-a871-4bc7597e373b","Type":"ContainerStarted","Data":"75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db"} Oct 30 06:18:17.693391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:17.693187 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" event={"ID":"4ef84b6f-f535-4625-a871-4bc7597e373b","Type":"ContainerStarted","Data":"005c31c5f417fc9820059927590d824ee59a44655f3344a8d0b39c541f4ec50a"} Oct 30 06:18:17.693391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:17.693216 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:17.738367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:17.738316 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" podStartSLOduration=1.738302764 podStartE2EDuration="1.738302764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:17.737849205 +0000 UTC m=+2804.313536259" watchObservedRunningTime="2026-10-30 06:18:17.738302764 +0000 UTC m=+2804.313989809" Oct 30 06:18:18.034560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:18.034486 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:18:18.034939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:18.034913 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:18:18.036124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:18.036093 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:18:20.341828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.341790 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" podUID="afa4b166-c592-4836-b7fc-d5548a148676" containerName="payload-processing" containerID="cri-o://c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467" gracePeriod=25 Oct 30 06:18:20.643434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.643408 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 06:18:20.664718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.664693 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume\") pod \"afa4b166-c592-4836-b7fc-d5548a148676\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " Oct 30 06:18:20.664818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.664803 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv\" (UniqueName: \"kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv\") pod \"afa4b166-c592-4836-b7fc-d5548a148676\" (UID: \"afa4b166-c592-4836-b7fc-d5548a148676\") " Oct 30 06:18:20.665039 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.665017 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume" pod "afa4b166-c592-4836-b7fc-d5548a148676" (UID: "afa4b166-c592-4836-b7fc-d5548a148676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:20.665341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.665315 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/afa4b166-c592-4836-b7fc-d5548a148676-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:20.666816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.666794 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv" pod "afa4b166-c592-4836-b7fc-d5548a148676" (UID: "afa4b166-c592-4836-b7fc-d5548a148676"). InnerVolumeSpecName "kube-api-access-r6mkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:20.720728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.720694 5565 generic.go:356] "Generic (PLEG): container finished" podID="afa4b166-c592-4836-b7fc-d5548a148676" containerID="c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467" exitCode=0 Oct 30 06:18:20.720878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.720756 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" Oct 30 06:18:20.720878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.720785 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" event={"ID":"afa4b166-c592-4836-b7fc-d5548a148676","Type":"ContainerDied","Data":"c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467"} Oct 30 06:18:20.720878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.720856 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4" event={"ID":"afa4b166-c592-4836-b7fc-d5548a148676","Type":"ContainerDied","Data":"812fe9852fe0fe29a8a3cc704f473528e8de72b9a5f3c8f871c23f98cbd145a6"} Oct 30 06:18:20.721087 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.720888 5565 scope.go:122] "RemoveContainer" containerID="c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467" Oct 30 06:18:20.733170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.733148 5565 scope.go:122] "RemoveContainer" containerID="c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467" Oct 30 06:18:20.733485 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:18:20.733463 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467\": container with ID starting with c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467 not found: ID does not exist" containerID="c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467" Oct 30 06:18:20.733617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.733488 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467"} err="failed to get container status \"c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467\": rpc error: code = NotFound desc = could not find container \"c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467\": container with ID starting with c2f2fbdc20734459de09324ee41dc0796e95bd9a278983f81134ef4306cc7467 not found: ID does not exist" Oct 30 06:18:20.766664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.766641 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6mkv\" (UniqueName: \"kubernetes.io/projected/afa4b166-c592-4836-b7fc-d5548a148676-kube-api-access-r6mkv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:20.786123 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.786097 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4"] Oct 30 06:18:20.789793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:20.789772 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d96a1f-5d988b98bb-8xrl4"] Oct 30 06:18:21.569123 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:21.569092 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afa4b166-c592-4836-b7fc-d5548a148676" path="/var/lib/kubelet/pods/afa4b166-c592-4836-b7fc-d5548a148676/volumes" Oct 30 06:18:24.915076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:24.915032 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm"] Oct 30 06:18:28.702938 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:28.702905 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:18:29.915843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:29.915801 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" podUID="f6849eb2-8b7b-4a87-a0df-e39d1903e41c" containerName="payload-pre-processing" containerID="cri-o://f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db" gracePeriod=25 Oct 30 06:18:30.207967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.207941 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:18:30.311419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.311364 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume\") pod \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " Oct 30 06:18:30.311627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.311517 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq\" (UniqueName: \"kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq\") pod \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\" (UID: \"f6849eb2-8b7b-4a87-a0df-e39d1903e41c\") " Oct 30 06:18:30.311837 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.311812 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume" pod "f6849eb2-8b7b-4a87-a0df-e39d1903e41c" (UID: "f6849eb2-8b7b-4a87-a0df-e39d1903e41c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:30.312059 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.312042 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:30.313624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.313597 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq" pod "f6849eb2-8b7b-4a87-a0df-e39d1903e41c" (UID: "f6849eb2-8b7b-4a87-a0df-e39d1903e41c"). InnerVolumeSpecName "kube-api-access-9qpkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:30.413155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.413119 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qpkq\" (UniqueName: \"kubernetes.io/projected/f6849eb2-8b7b-4a87-a0df-e39d1903e41c-kube-api-access-9qpkq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:30.798741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.798703 5565 generic.go:356] "Generic (PLEG): container finished" podID="f6849eb2-8b7b-4a87-a0df-e39d1903e41c" containerID="f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db" exitCode=0 Oct 30 06:18:30.798913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.798751 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" Oct 30 06:18:30.798913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.798749 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" event={"ID":"f6849eb2-8b7b-4a87-a0df-e39d1903e41c","Type":"ContainerDied","Data":"f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db"} Oct 30 06:18:30.798913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.798858 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm" event={"ID":"f6849eb2-8b7b-4a87-a0df-e39d1903e41c","Type":"ContainerDied","Data":"9b80b8bd7d253870b2fae77766439e8b89cf56883218eaf5d007dc7e539e35cb"} Oct 30 06:18:30.798913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.798887 5565 scope.go:122] "RemoveContainer" containerID="f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db" Oct 30 06:18:30.809504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.809483 5565 scope.go:122] "RemoveContainer" containerID="f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db" Oct 30 06:18:30.809896 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:18:30.809868 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db\": container with ID starting with f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db not found: ID does not exist" containerID="f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db" Oct 30 06:18:30.810007 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.809904 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db"} err="failed to get container status \"f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db\": rpc error: code = NotFound desc = could not find container \"f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db\": container with ID starting with f810bbc5c0cccd78393ad6ae6bbf7a7bf10925c4a52e918bef68d31a63fd22db not found: ID does not exist" Oct 30 06:18:30.857706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.857683 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm"] Oct 30 06:18:30.871639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:30.871609 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-07b278-99f7dd4b7-7krdm"] Oct 30 06:18:31.568961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:31.568916 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6849eb2-8b7b-4a87-a0df-e39d1903e41c" path="/var/lib/kubelet/pods/f6849eb2-8b7b-4a87-a0df-e39d1903e41c/volumes" Oct 30 06:18:34.692552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:34.680441 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:18:36.677427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:36.677387 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x"] Oct 30 06:18:36.690565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:36.690518 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-d96a1f-glc9x"] Oct 30 06:18:37.569139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:37.569095 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b" path="/var/lib/kubelet/pods/7b2e4f4f-dd7e-434a-91aa-ac5f14727a1b/volumes" Oct 30 06:18:39.025988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.012645 5565 scope.go:122] "RemoveContainer" containerID="f062d8e88c1ac5b35c267974064cfeb20ec76dff0b3a2f675b54fa38b9d0b1f9" Oct 30 06:18:39.629393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.629354 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp"] Oct 30 06:18:39.630278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630253 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f6849eb2-8b7b-4a87-a0df-e39d1903e41c" containerName="payload-pre-processing" Oct 30 06:18:39.630278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630276 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6849eb2-8b7b-4a87-a0df-e39d1903e41c" containerName="payload-pre-processing" Oct 30 06:18:39.630424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630308 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" Oct 30 06:18:39.630424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630313 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d078081e-660d-4c3a-8c14-dd61d34f9e82" containerName="istio-proxy" Oct 30 06:18:39.630424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630330 5565 cpu_manager.go:414] "containerMap: removing container" podUID="afa4b166-c592-4836-b7fc-d5548a148676" containerName="payload-processing" Oct 30 06:18:39.630424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630335 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afa4b166-c592-4836-b7fc-d5548a148676" containerName="payload-processing" Oct 30 06:18:39.630606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630434 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="afa4b166-c592-4836-b7fc-d5548a148676" containerName="payload-processing" Oct 30 06:18:39.630606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.630449 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6849eb2-8b7b-4a87-a0df-e39d1903e41c" containerName="payload-pre-processing" Oct 30 06:18:39.635844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.635825 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.652842 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.652799 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8b892\"" Oct 30 06:18:39.652956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.652841 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8b892-dockercfg-zbh5v\"" Oct 30 06:18:39.653980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.653958 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp"] Oct 30 06:18:39.701370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.701344 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84"] Oct 30 06:18:39.706736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.706718 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:39.724097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.724072 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84"] Oct 30 06:18:39.750230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.747760 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggnt2\" (UniqueName: \"kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2\") pod \"payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.750230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.748012 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.851548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.850541 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.851548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.850713 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr6ff\" (UniqueName: \"kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:39.851548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.850804 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggnt2\" (UniqueName: \"kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2\") pod \"payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.851548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.850961 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:39.882594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.881969 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.946968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.946937 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggnt2\" (UniqueName: \"kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2\") pod \"payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.949929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.949893 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:39.968549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.960123 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr6ff\" (UniqueName: \"kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:39.968549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.960333 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:39.968549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.964115 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:39.990032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:39.989880 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr6ff\" (UniqueName: \"kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:40.020859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.020353 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:40.286777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.286752 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp"] Oct 30 06:18:40.287799 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:18:40.287770 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd0eaa1e_aca6_4b5d_abe2_a55342f64702.slice/crio-2ef0bf8f143dd84a18cebc3e1d24c453f37bc10fb6bb7db200438f7604697fa6 WatchSource:0}: Error finding container 2ef0bf8f143dd84a18cebc3e1d24c453f37bc10fb6bb7db200438f7604697fa6: Status 404 returned error can't find the container with id 2ef0bf8f143dd84a18cebc3e1d24c453f37bc10fb6bb7db200438f7604697fa6 Oct 30 06:18:40.382387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.382364 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84"] Oct 30 06:18:40.382947 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:18:40.382917 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd677b6c_37b8_44b3_8e70_3d25048be781.slice/crio-f9a60cde443bcb696d6043f34efe087664a825db5b4796bc54b47607aa89c17c WatchSource:0}: Error finding container f9a60cde443bcb696d6043f34efe087664a825db5b4796bc54b47607aa89c17c: Status 404 returned error can't find the container with id f9a60cde443bcb696d6043f34efe087664a825db5b4796bc54b47607aa89c17c Oct 30 06:18:40.960540 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.960490 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" event={"ID":"cd0eaa1e-aca6-4b5d-abe2-a55342f64702","Type":"ContainerStarted","Data":"4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49"} Oct 30 06:18:40.960741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.960591 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" event={"ID":"cd0eaa1e-aca6-4b5d-abe2-a55342f64702","Type":"ContainerStarted","Data":"2ef0bf8f143dd84a18cebc3e1d24c453f37bc10fb6bb7db200438f7604697fa6"} Oct 30 06:18:40.960741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.960630 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:18:40.962369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.962334 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" event={"ID":"cd677b6c-37b8-44b3-8e70-3d25048be781","Type":"ContainerStarted","Data":"177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f"} Oct 30 06:18:40.962481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.962387 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" event={"ID":"cd677b6c-37b8-44b3-8e70-3d25048be781","Type":"ContainerStarted","Data":"f9a60cde443bcb696d6043f34efe087664a825db5b4796bc54b47607aa89c17c"} Oct 30 06:18:40.962481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:40.962413 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:41.007678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:41.007635 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" podStartSLOduration=2.007621832 podStartE2EDuration="2.007621832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:41.005272346 +0000 UTC m=+2827.580959416" watchObservedRunningTime="2026-10-30 06:18:41.007621832 +0000 UTC m=+2827.583308885" Oct 30 06:18:41.052168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:41.052127 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" podStartSLOduration=2.052114251 podStartE2EDuration="2.052114251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:18:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:18:41.04950553 +0000 UTC m=+2827.625192585" watchObservedRunningTime="2026-10-30 06:18:41.052114251 +0000 UTC m=+2827.627801305" Oct 30 06:18:42.947971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:42.947940 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd_b4a2fd88-6c33-4ebf-afba-0f8a87720e92/payload-processing/0.log" Oct 30 06:18:43.136893 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:43.136867 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:18:43.339751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:43.339717 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:18:43.534947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:43.534918 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7_baaa1333-908d-4961-8812-675996545881/payload-processing/0.log" Oct 30 06:18:43.733644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:43.728829 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7_baaa1333-908d-4961-8812-675996545881/payload-processing/0.log" Oct 30 06:18:44.016431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.016403 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj"] Oct 30 06:18:44.021287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.021269 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.040924 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.040898 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj"] Oct 30 06:18:44.069197 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.069172 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:18:44.069329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.069247 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:18:44.137180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.137144 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.137401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.137382 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.137480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.137437 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2p57\" (UniqueName: \"kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.238748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.238722 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.238902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.238767 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2p57\" (UniqueName: \"kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.238902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.238837 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.239358 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.239336 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.241014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.240992 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.263917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.263888 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2p57\" (UniqueName: \"kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.338228 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.338161 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:44.647240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:44.647210 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj"] Oct 30 06:18:44.648391 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:18:44.648367 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8965c9d1_ec0a_4da4_911a_cb2fb2e820be.slice/crio-5ea8dc8b0d7ee9a68bb5b235463144e42bd4fb6168f9341796044e5c82706c7f WatchSource:0}: Error finding container 5ea8dc8b0d7ee9a68bb5b235463144e42bd4fb6168f9341796044e5c82706c7f: Status 404 returned error can't find the container with id 5ea8dc8b0d7ee9a68bb5b235463144e42bd4fb6168f9341796044e5c82706c7f Oct 30 06:18:45.043305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:45.043272 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" event={"ID":"8965c9d1-ec0a-4da4-911a-cb2fb2e820be","Type":"ContainerStarted","Data":"5ea8dc8b0d7ee9a68bb5b235463144e42bd4fb6168f9341796044e5c82706c7f"} Oct 30 06:18:46.052795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:46.052757 5565 generic.go:356] "Generic (PLEG): container finished" podID="8965c9d1-ec0a-4da4-911a-cb2fb2e820be" containerID="684308037d5d7aee3f1aed2493e6549034d72ef3a5bcd39330b5c9f380b9531a" exitCode=0 Oct 30 06:18:46.053242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:46.052805 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" event={"ID":"8965c9d1-ec0a-4da4-911a-cb2fb2e820be","Type":"ContainerDied","Data":"684308037d5d7aee3f1aed2493e6549034d72ef3a5bcd39330b5c9f380b9531a"} Oct 30 06:18:47.229668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.229637 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:47.382062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.381984 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca\") pod \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " Oct 30 06:18:47.382270 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.382250 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca\") pod \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " Oct 30 06:18:47.382352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.382289 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57\" (UniqueName: \"kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57\") pod \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\" (UID: \"8965c9d1-ec0a-4da4-911a-cb2fb2e820be\") " Oct 30 06:18:47.382650 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.382623 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca" pod "8965c9d1-ec0a-4da4-911a-cb2fb2e820be" (UID: "8965c9d1-ec0a-4da4-911a-cb2fb2e820be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:47.383065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.383040 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:47.384619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.384594 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca" pod "8965c9d1-ec0a-4da4-911a-cb2fb2e820be" (UID: "8965c9d1-ec0a-4da4-911a-cb2fb2e820be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:18:47.385236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.385219 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57" pod "8965c9d1-ec0a-4da4-911a-cb2fb2e820be" (UID: "8965c9d1-ec0a-4da4-911a-cb2fb2e820be"). InnerVolumeSpecName "kube-api-access-f2p57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:47.484997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.484963 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:47.484997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.484990 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2p57\" (UniqueName: \"kubernetes.io/projected/8965c9d1-ec0a-4da4-911a-cb2fb2e820be-kube-api-access-f2p57\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:47.534282 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:18:47.534251 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:18:47.985396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.985361 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm"] Oct 30 06:18:47.985628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:47.985579 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" podUID="885d6370-e620-4728-af2d-37dc726d58bc" containerName="payload-processing" containerID="cri-o://a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275" gracePeriod=30 Oct 30 06:18:48.024537 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.024494 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb"] Oct 30 06:18:48.024779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.024753 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" podUID="0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" containerName="payload-pre-processing" containerID="cri-o://f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528" gracePeriod=30 Oct 30 06:18:48.076442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.076407 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" event={"ID":"8965c9d1-ec0a-4da4-911a-cb2fb2e820be","Type":"ContainerDied","Data":"5ea8dc8b0d7ee9a68bb5b235463144e42bd4fb6168f9341796044e5c82706c7f"} Oct 30 06:18:48.076616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.076457 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj" Oct 30 06:18:48.076616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.076466 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ea8dc8b0d7ee9a68bb5b235463144e42bd4fb6168f9341796044e5c82706c7f" Oct 30 06:18:48.549340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.549005 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:18:48.552341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.552311 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:18:48.647019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.646986 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt\" (UniqueName: \"kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt\") pod \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " Oct 30 06:18:48.647267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.647149 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume\") pod \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\" (UID: \"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8\") " Oct 30 06:18:48.647267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.647193 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume\") pod \"885d6370-e620-4728-af2d-37dc726d58bc\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " Oct 30 06:18:48.647694 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.647666 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume" pod "0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" (UID: "0d16eafb-cedd-4a48-9e6e-77b7bf42efd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:48.647778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.647670 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume" pod "885d6370-e620-4728-af2d-37dc726d58bc" (UID: "885d6370-e620-4728-af2d-37dc726d58bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:18:48.647842 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.647796 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66\") pod \"885d6370-e620-4728-af2d-37dc726d58bc\" (UID: \"885d6370-e620-4728-af2d-37dc726d58bc\") " Oct 30 06:18:48.648581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.648553 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:48.648581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.648582 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/885d6370-e620-4728-af2d-37dc726d58bc-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:48.649789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.649758 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt" pod "0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" (UID: "0d16eafb-cedd-4a48-9e6e-77b7bf42efd8"). InnerVolumeSpecName "kube-api-access-hj5bt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:48.650216 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.650192 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66" pod "885d6370-e620-4728-af2d-37dc726d58bc" (UID: "885d6370-e620-4728-af2d-37dc726d58bc"). InnerVolumeSpecName "kube-api-access-8xj66". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:18:48.705434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.705405 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg"] Oct 30 06:18:48.711956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.711930 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-ebe61e-787pg"] Oct 30 06:18:48.750389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.750312 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xj66\" (UniqueName: \"kubernetes.io/projected/885d6370-e620-4728-af2d-37dc726d58bc-kube-api-access-8xj66\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:48.750389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:48.750340 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj5bt\" (UniqueName: \"kubernetes.io/projected/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8-kube-api-access-hj5bt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:18:49.085308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.085274 5565 generic.go:356] "Generic (PLEG): container finished" podID="885d6370-e620-4728-af2d-37dc726d58bc" containerID="a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275" exitCode=0 Oct 30 06:18:49.085781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.085319 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" Oct 30 06:18:49.085781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.085359 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" event={"ID":"885d6370-e620-4728-af2d-37dc726d58bc","Type":"ContainerDied","Data":"a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275"} Oct 30 06:18:49.085781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.085415 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm" event={"ID":"885d6370-e620-4728-af2d-37dc726d58bc","Type":"ContainerDied","Data":"71d329704b929c2c5f899ee62604e0da05313192d7caa6e105b315d5dcc5428c"} Oct 30 06:18:49.085781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.085446 5565 scope.go:122] "RemoveContainer" containerID="a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275" Oct 30 06:18:49.086995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.086965 5565 generic.go:356] "Generic (PLEG): container finished" podID="0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" containerID="f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528" exitCode=0 Oct 30 06:18:49.087120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.086994 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" event={"ID":"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8","Type":"ContainerDied","Data":"f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528"} Oct 30 06:18:49.087120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.087006 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" Oct 30 06:18:49.087120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.087045 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb" event={"ID":"0d16eafb-cedd-4a48-9e6e-77b7bf42efd8","Type":"ContainerDied","Data":"6c6348c3d44fac676d4e74216044676b646e23a461775874df5d775c0d57a81a"} Oct 30 06:18:49.100790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.100770 5565 scope.go:122] "RemoveContainer" containerID="a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275" Oct 30 06:18:49.101061 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:18:49.101042 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275\": container with ID starting with a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275 not found: ID does not exist" containerID="a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275" Oct 30 06:18:49.101120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.101069 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275"} err="failed to get container status \"a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275\": rpc error: code = NotFound desc = could not find container \"a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275\": container with ID starting with a9b7bbd00887ee8d6566bd2739518a36124e5fc4ca929978bdb42899f6a3d275 not found: ID does not exist" Oct 30 06:18:49.101120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.101089 5565 scope.go:122] "RemoveContainer" containerID="f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528" Oct 30 06:18:49.110957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.110940 5565 scope.go:122] "RemoveContainer" containerID="f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528" Oct 30 06:18:49.111220 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:18:49.111203 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528\": container with ID starting with f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528 not found: ID does not exist" containerID="f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528" Oct 30 06:18:49.111274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.111236 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528"} err="failed to get container status \"f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528\": rpc error: code = NotFound desc = could not find container \"f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528\": container with ID starting with f08d78ffe158b7eb90e310b615cdd4c2bea0c94f7b63a3bdafbd12c69a2f3528 not found: ID does not exist" Oct 30 06:18:49.143497 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.143476 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm"] Oct 30 06:18:49.148388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.148362 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-ebe61e-557897d54-cg7lm"] Oct 30 06:18:49.185347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.185322 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb"] Oct 30 06:18:49.191428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.191409 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-ebe61e-78bdfc88f9-mdkjb"] Oct 30 06:18:49.569956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.569925 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" path="/var/lib/kubelet/pods/0d16eafb-cedd-4a48-9e6e-77b7bf42efd8/volumes" Oct 30 06:18:49.570384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.570370 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="302ab598-206a-41a5-a026-12229cf1469b" path="/var/lib/kubelet/pods/302ab598-206a-41a5-a026-12229cf1469b/volumes" Oct 30 06:18:49.570798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:49.570783 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="885d6370-e620-4728-af2d-37dc726d58bc" path="/var/lib/kubelet/pods/885d6370-e620-4728-af2d-37dc726d58bc/volumes" Oct 30 06:18:51.973288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:51.973257 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:18:51.973744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:18:51.973628 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:19:08.498403 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.498368 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6"] Oct 30 06:19:08.499568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499510 5565 cpu_manager.go:414] "containerMap: removing container" podUID="885d6370-e620-4728-af2d-37dc726d58bc" containerName="payload-processing" Oct 30 06:19:08.499568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499546 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="885d6370-e620-4728-af2d-37dc726d58bc" containerName="payload-processing" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499585 5565 cpu_manager.go:414] "containerMap: removing container" podUID="8965c9d1-ec0a-4da4-911a-cb2fb2e820be" containerName="revoke-keys" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499593 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8965c9d1-ec0a-4da4-911a-cb2fb2e820be" containerName="revoke-keys" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499634 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" containerName="payload-pre-processing" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499642 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" containerName="payload-pre-processing" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499810 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="885d6370-e620-4728-af2d-37dc726d58bc" containerName="payload-processing" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499824 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="8965c9d1-ec0a-4da4-911a-cb2fb2e820be" containerName="revoke-keys" Oct 30 06:19:08.499836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.499838 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d16eafb-cedd-4a48-9e6e-77b7bf42efd8" containerName="payload-pre-processing" Oct 30 06:19:08.504003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.503970 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.519390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.519364 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6"] Oct 30 06:19:08.525935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.525909 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k98bh\" (UniqueName: \"kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh\") pod \"payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.526062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.525967 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.528697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.528671 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-07b278\"" Oct 30 06:19:08.529054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.529033 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-07b278-dockercfg-cjm6p\"" Oct 30 06:19:08.627342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.627311 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k98bh\" (UniqueName: \"kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh\") pod \"payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.627494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.627372 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.628239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.628218 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.653073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.653046 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k98bh\" (UniqueName: \"kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh\") pod \"payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:08.823178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:08.823153 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:09.133288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:09.133117 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6"] Oct 30 06:19:09.133425 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:19:09.133329 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68f9db15_ee0f_4486_b455_d63e690d9e78.slice/crio-43724d03642a139724e738cf416110c0d4b6bfc5b37fafbff1e046e4915f0de0 WatchSource:0}: Error finding container 43724d03642a139724e738cf416110c0d4b6bfc5b37fafbff1e046e4915f0de0: Status 404 returned error can't find the container with id 43724d03642a139724e738cf416110c0d4b6bfc5b37fafbff1e046e4915f0de0 Oct 30 06:19:09.349851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:09.349815 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" event={"ID":"68f9db15-ee0f-4486-b455-d63e690d9e78","Type":"ContainerStarted","Data":"b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0"} Oct 30 06:19:09.350036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:09.349877 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" event={"ID":"68f9db15-ee0f-4486-b455-d63e690d9e78","Type":"ContainerStarted","Data":"43724d03642a139724e738cf416110c0d4b6bfc5b37fafbff1e046e4915f0de0"} Oct 30 06:19:09.350645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:09.350607 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:09.398185 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:09.398103 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" podStartSLOduration=1.398091319 podStartE2EDuration="1.398091319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:09.397741546 +0000 UTC m=+2855.973428591" watchObservedRunningTime="2026-10-30 06:19:09.398091319 +0000 UTC m=+2855.973778373" Oct 30 06:19:17.534460 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:19:17.534414 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:19:21.369557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:21.369515 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:19:32.571764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:32.571727 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm"] Oct 30 06:19:32.636323 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:32.636283 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" podUID="599547ed-d597-42b6-a585-bec48f04059e" containerName="maas-api" containerID="cri-o://98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a" gracePeriod=30 Oct 30 06:19:33.171522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.171497 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 06:19:33.270981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.270945 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m\" (UniqueName: \"kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m\") pod \"599547ed-d597-42b6-a585-bec48f04059e\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " Oct 30 06:19:33.271181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.271114 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca\") pod \"599547ed-d597-42b6-a585-bec48f04059e\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " Oct 30 06:19:33.271267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.271239 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls\") pod \"599547ed-d597-42b6-a585-bec48f04059e\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " Oct 30 06:19:33.271358 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.271336 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs\") pod \"599547ed-d597-42b6-a585-bec48f04059e\" (UID: \"599547ed-d597-42b6-a585-bec48f04059e\") " Oct 30 06:19:33.271506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.271482 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca" pod "599547ed-d597-42b6-a585-bec48f04059e" (UID: "599547ed-d597-42b6-a585-bec48f04059e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:33.271988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.271966 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/599547ed-d597-42b6-a585-bec48f04059e-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:33.273774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.273746 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m" pod "599547ed-d597-42b6-a585-bec48f04059e" (UID: "599547ed-d597-42b6-a585-bec48f04059e"). InnerVolumeSpecName "kube-api-access-h4w8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:33.273893 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.273847 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls" pod "599547ed-d597-42b6-a585-bec48f04059e" (UID: "599547ed-d597-42b6-a585-bec48f04059e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:33.274242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.274212 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs" pod "599547ed-d597-42b6-a585-bec48f04059e" (UID: "599547ed-d597-42b6-a585-bec48f04059e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:33.373141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.373115 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:33.373141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.373140 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4w8m\" (UniqueName: \"kubernetes.io/projected/599547ed-d597-42b6-a585-bec48f04059e-kube-api-access-h4w8m\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:33.373307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.373151 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/599547ed-d597-42b6-a585-bec48f04059e-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:33.390988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.390921 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj"] Oct 30 06:19:33.394380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.394353 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-37bbcc-r9lhj"] Oct 30 06:19:33.569955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.569922 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8965c9d1-ec0a-4da4-911a-cb2fb2e820be" path="/var/lib/kubelet/pods/8965c9d1-ec0a-4da4-911a-cb2fb2e820be/volumes" Oct 30 06:19:33.644609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.644510 5565 generic.go:356] "Generic (PLEG): container finished" podID="599547ed-d597-42b6-a585-bec48f04059e" containerID="98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a" exitCode=0 Oct 30 06:19:33.644609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.644590 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" Oct 30 06:19:33.644609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.644594 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" event={"ID":"599547ed-d597-42b6-a585-bec48f04059e","Type":"ContainerDied","Data":"98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a"} Oct 30 06:19:33.645067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.644655 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm" event={"ID":"599547ed-d597-42b6-a585-bec48f04059e","Type":"ContainerDied","Data":"1ad0aeb7222417206541c466788af5271c44f32d31b689dc9bd6e83bc4647d05"} Oct 30 06:19:33.645067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.644678 5565 scope.go:122] "RemoveContainer" containerID="98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a" Oct 30 06:19:33.655461 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.655442 5565 scope.go:122] "RemoveContainer" containerID="98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a" Oct 30 06:19:33.655737 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:19:33.655719 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a\": container with ID starting with 98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a not found: ID does not exist" containerID="98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a" Oct 30 06:19:33.655809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.655745 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a"} err="failed to get container status \"98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a\": rpc error: code = NotFound desc = could not find container \"98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a\": container with ID starting with 98975b6ae6a56e2abc71c19f7b1a07cd7e3b709fe8b6e5a7ae7a406e7aae803a not found: ID does not exist" Oct 30 06:19:33.709664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.709636 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm"] Oct 30 06:19:33.718718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:33.718692 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-37bbcc-b8f6f6ffd-cm9dm"] Oct 30 06:19:35.569045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:35.569011 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="599547ed-d597-42b6-a585-bec48f04059e" path="/var/lib/kubelet/pods/599547ed-d597-42b6-a585-bec48f04059e/volumes" Oct 30 06:19:38.141712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:38.141668 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:19:40.970720 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:40.970675 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962"] Oct 30 06:19:40.979103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:40.979071 5565 cpu_manager.go:414] "containerMap: removing container" podUID="599547ed-d597-42b6-a585-bec48f04059e" containerName="maas-api" Oct 30 06:19:40.979364 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:40.979346 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="599547ed-d597-42b6-a585-bec48f04059e" containerName="maas-api" Oct 30 06:19:40.979764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:40.979744 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="599547ed-d597-42b6-a585-bec48f04059e" containerName="maas-api" Oct 30 06:19:41.050169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.050134 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962"] Oct 30 06:19:41.050609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.050497 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.069724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.069697 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-7fdb95\"" Oct 30 06:19:41.124376 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.124348 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2"] Oct 30 06:19:41.128759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.128740 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.151106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.151080 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2"] Oct 30 06:19:41.198423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.198400 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-7fdb95-dockercfg-6lq7w\"" Oct 30 06:19:41.209579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.209552 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdcnc\" (UniqueName: \"kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.209721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.209680 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.212814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.212793 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67"] Oct 30 06:19:41.217174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.217158 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.238123 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.238100 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67"] Oct 30 06:19:41.299736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.299714 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:41.300034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.300016 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:19:41.311144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311117 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.311264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311168 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tzhcr\" (UniqueName: \"kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.311264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311255 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fb8r\" (UniqueName: \"kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r\") pod \"payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.311414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311394 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.311481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311428 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.311481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311471 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdcnc\" (UniqueName: \"kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.311638 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311615 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.311874 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.311856 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.338220 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.338188 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdcnc\" (UniqueName: \"kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc\") pod \"payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.372624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.372598 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.412709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.412678 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.412852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.412718 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.412946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.412919 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.413045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.413031 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tzhcr\" (UniqueName: \"kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.413114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.413102 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fb8r\" (UniqueName: \"kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r\") pod \"payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.414085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.414039 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.414453 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.414429 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.415637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.415613 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.445513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.445472 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tzhcr\" (UniqueName: \"kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr\") pod \"maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.453847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.453793 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fb8r\" (UniqueName: \"kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r\") pod \"payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.532038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.532008 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:41.701523 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:19:41.701483 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca564ba7_731b_4d1d_b155_1dd05c326809.slice/crio-e2c08ad2ff2b724547c61ad6aee01fb8f952a76aa7902269822e07f8f6f3a882 WatchSource:0}: Error finding container e2c08ad2ff2b724547c61ad6aee01fb8f952a76aa7902269822e07f8f6f3a882: Status 404 returned error can't find the container with id e2c08ad2ff2b724547c61ad6aee01fb8f952a76aa7902269822e07f8f6f3a882 Oct 30 06:19:41.702053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.702032 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962"] Oct 30 06:19:41.743345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.743316 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:41.837340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.837306 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" event={"ID":"ca564ba7-731b-4d1d-b155-1dd05c326809","Type":"ContainerStarted","Data":"930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692"} Oct 30 06:19:41.837520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.837368 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" event={"ID":"ca564ba7-731b-4d1d-b155-1dd05c326809","Type":"ContainerStarted","Data":"e2c08ad2ff2b724547c61ad6aee01fb8f952a76aa7902269822e07f8f6f3a882"} Oct 30 06:19:41.837645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.837580 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:41.883418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.883371 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" podStartSLOduration=1.883357377 podStartE2EDuration="1.883357377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:41.881784715 +0000 UTC m=+2888.457471771" watchObservedRunningTime="2026-10-30 06:19:41.883357377 +0000 UTC m=+2888.459044452" Oct 30 06:19:41.897062 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:19:41.897028 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddea1b9f7_4502_4c29_b6ac_41309e2a5113.slice/crio-5d86ac3a4e3c77f8cb5896a587e3f4eb65d636df0cb5dbbb68c046b821ba5721 WatchSource:0}: Error finding container 5d86ac3a4e3c77f8cb5896a587e3f4eb65d636df0cb5dbbb68c046b821ba5721: Status 404 returned error can't find the container with id 5d86ac3a4e3c77f8cb5896a587e3f4eb65d636df0cb5dbbb68c046b821ba5721 Oct 30 06:19:41.899178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:41.899150 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67"] Oct 30 06:19:42.071427 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:19:42.071394 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb159bb4f_a575_4cae_8dc4_e6f5582fc629.slice/crio-61556bbd6586212321484f7d7054be19c7e7bb441ff33578223837c4e4a6b5e2 WatchSource:0}: Error finding container 61556bbd6586212321484f7d7054be19c7e7bb441ff33578223837c4e4a6b5e2: Status 404 returned error can't find the container with id 61556bbd6586212321484f7d7054be19c7e7bb441ff33578223837c4e4a6b5e2 Oct 30 06:19:42.072013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.071988 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2"] Oct 30 06:19:42.847291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.847254 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" event={"ID":"b159bb4f-a575-4cae-8dc4-e6f5582fc629","Type":"ContainerStarted","Data":"f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63"} Oct 30 06:19:42.847436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.847319 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" event={"ID":"b159bb4f-a575-4cae-8dc4-e6f5582fc629","Type":"ContainerStarted","Data":"61556bbd6586212321484f7d7054be19c7e7bb441ff33578223837c4e4a6b5e2"} Oct 30 06:19:42.847436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.847350 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:42.848945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.848922 5565 generic.go:356] "Generic (PLEG): container finished" podID="dea1b9f7-4502-4c29-b6ac-41309e2a5113" containerID="1d8e7040cb396ae62ef1dda07d64d9aeb8890c8bcf30390d7a18ca4c39e76a1f" exitCode=0 Oct 30 06:19:42.849048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.848997 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" event={"ID":"dea1b9f7-4502-4c29-b6ac-41309e2a5113","Type":"ContainerDied","Data":"1d8e7040cb396ae62ef1dda07d64d9aeb8890c8bcf30390d7a18ca4c39e76a1f"} Oct 30 06:19:42.849048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.849039 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" event={"ID":"dea1b9f7-4502-4c29-b6ac-41309e2a5113","Type":"ContainerStarted","Data":"5d86ac3a4e3c77f8cb5896a587e3f4eb65d636df0cb5dbbb68c046b821ba5721"} Oct 30 06:19:42.893705 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:42.893664 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" podStartSLOduration=1.89365087 podStartE2EDuration="1.89365087s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:42.891826098 +0000 UTC m=+2889.467513152" watchObservedRunningTime="2026-10-30 06:19:42.89365087 +0000 UTC m=+2889.469337924" Oct 30 06:19:44.013771 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.013749 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:44.066324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.066294 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca\") pod \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " Oct 30 06:19:44.066475 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.066455 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca\") pod \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " Oct 30 06:19:44.066572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.066554 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr\" (UniqueName: \"kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr\") pod \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\" (UID: \"dea1b9f7-4502-4c29-b6ac-41309e2a5113\") " Oct 30 06:19:44.066654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.066629 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca" pod "dea1b9f7-4502-4c29-b6ac-41309e2a5113" (UID: "dea1b9f7-4502-4c29-b6ac-41309e2a5113"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:44.067206 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.067180 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:44.068684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.068657 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca" pod "dea1b9f7-4502-4c29-b6ac-41309e2a5113" (UID: "dea1b9f7-4502-4c29-b6ac-41309e2a5113"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:44.068684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.068667 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr" pod "dea1b9f7-4502-4c29-b6ac-41309e2a5113" (UID: "dea1b9f7-4502-4c29-b6ac-41309e2a5113"). InnerVolumeSpecName "kube-api-access-tzhcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:44.169154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.169088 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dea1b9f7-4502-4c29-b6ac-41309e2a5113-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:44.169154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.169115 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tzhcr\" (UniqueName: \"kubernetes.io/projected/dea1b9f7-4502-4c29-b6ac-41309e2a5113-kube-api-access-tzhcr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:44.866031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.865996 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" Oct 30 06:19:44.866031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.866022 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67" event={"ID":"dea1b9f7-4502-4c29-b6ac-41309e2a5113","Type":"ContainerDied","Data":"5d86ac3a4e3c77f8cb5896a587e3f4eb65d636df0cb5dbbb68c046b821ba5721"} Oct 30 06:19:44.866252 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:44.866073 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d86ac3a4e3c77f8cb5896a587e3f4eb65d636df0cb5dbbb68c046b821ba5721" Oct 30 06:19:45.997847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:45.997811 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq"] Oct 30 06:19:46.011909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:46.011882 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n"] Oct 30 06:19:47.533798 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:19:47.533756 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:19:51.574208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:51.574176 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v"] Oct 30 06:19:51.608268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:51.608232 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b"] Oct 30 06:19:52.852050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:52.852021 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:19:53.063589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.063551 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk"] Oct 30 06:19:53.063783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.063746 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" podUID="4fbb7b13-671d-4383-bef4-2cf446b2e99b" containerName="payload-processing" containerID="cri-o://5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab" gracePeriod=30 Oct 30 06:19:53.086554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.086505 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv"] Oct 30 06:19:53.088308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.086747 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" podUID="70ea4097-fdd9-4811-aa82-1b5d8423de58" containerName="payload-pre-processing" containerID="cri-o://0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91" gracePeriod=30 Oct 30 06:19:53.498554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.481969 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:19:53.525854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.525821 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m"] Oct 30 06:19:53.528096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.528035 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4fbb7b13-671d-4383-bef4-2cf446b2e99b" containerName="payload-processing" Oct 30 06:19:53.528273 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.528256 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4fbb7b13-671d-4383-bef4-2cf446b2e99b" containerName="payload-processing" Oct 30 06:19:53.528444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.528433 5565 cpu_manager.go:414] "containerMap: removing container" podUID="dea1b9f7-4502-4c29-b6ac-41309e2a5113" containerName="revoke-keys" Oct 30 06:19:53.528545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.528519 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dea1b9f7-4502-4c29-b6ac-41309e2a5113" containerName="revoke-keys" Oct 30 06:19:53.528939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.528922 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="dea1b9f7-4502-4c29-b6ac-41309e2a5113" containerName="revoke-keys" Oct 30 06:19:53.529072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.529060 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4fbb7b13-671d-4383-bef4-2cf446b2e99b" containerName="payload-processing" Oct 30 06:19:53.546199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.546174 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.572942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.572903 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:19:53.573248 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.573230 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:19:53.594549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.592201 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m"] Oct 30 06:19:53.620671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.620639 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:19:53.652401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.652333 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l\" (UniqueName: \"kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l\") pod \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " Oct 30 06:19:53.652610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.652590 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume\") pod \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\" (UID: \"4fbb7b13-671d-4383-bef4-2cf446b2e99b\") " Oct 30 06:19:53.652969 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.652935 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume" pod "4fbb7b13-671d-4383-bef4-2cf446b2e99b" (UID: "4fbb7b13-671d-4383-bef4-2cf446b2e99b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:53.653327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.653307 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.653423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.653344 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m9kd\" (UniqueName: \"kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.653487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.653443 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.653566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.653548 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4fbb7b13-671d-4383-bef4-2cf446b2e99b-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.654690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.654667 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l" pod "4fbb7b13-671d-4383-bef4-2cf446b2e99b" (UID: "4fbb7b13-671d-4383-bef4-2cf446b2e99b"). InnerVolumeSpecName "kube-api-access-sg58l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:53.754213 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.754185 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume\") pod \"70ea4097-fdd9-4811-aa82-1b5d8423de58\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " Oct 30 06:19:53.754363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.754270 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498\" (UniqueName: \"kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498\") pod \"70ea4097-fdd9-4811-aa82-1b5d8423de58\" (UID: \"70ea4097-fdd9-4811-aa82-1b5d8423de58\") " Oct 30 06:19:53.754602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.754582 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume" pod "70ea4097-fdd9-4811-aa82-1b5d8423de58" (UID: "70ea4097-fdd9-4811-aa82-1b5d8423de58"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:53.754724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.754701 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.754843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.754740 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m9kd\" (UniqueName: \"kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.754843 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.754776 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.755030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.755013 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70ea4097-fdd9-4811-aa82-1b5d8423de58-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.755080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.755038 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg58l\" (UniqueName: \"kubernetes.io/projected/4fbb7b13-671d-4383-bef4-2cf446b2e99b-kube-api-access-sg58l\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.755391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.755372 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.756565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.756544 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498" pod "70ea4097-fdd9-4811-aa82-1b5d8423de58" (UID: "70ea4097-fdd9-4811-aa82-1b5d8423de58"). InnerVolumeSpecName "kube-api-access-2m498". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:53.757073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.757056 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.780031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.780010 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m9kd\" (UniqueName: \"kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd\") pod \"maas-api-revoke-sub-a47c8008-6w59m\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.862931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.862882 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2m498\" (UniqueName: \"kubernetes.io/projected/70ea4097-fdd9-4811-aa82-1b5d8423de58-kube-api-access-2m498\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:53.867577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.867512 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:19:53.902308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.902279 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:53.941032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.940999 5565 generic.go:356] "Generic (PLEG): container finished" podID="70ea4097-fdd9-4811-aa82-1b5d8423de58" containerID="0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91" exitCode=0 Oct 30 06:19:53.941224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.941210 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" Oct 30 06:19:53.942138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.941818 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" event={"ID":"70ea4097-fdd9-4811-aa82-1b5d8423de58","Type":"ContainerDied","Data":"0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91"} Oct 30 06:19:53.942138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.941883 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv" event={"ID":"70ea4097-fdd9-4811-aa82-1b5d8423de58","Type":"ContainerDied","Data":"53cfc4706b5849d118852b61499464a11d8d5ddcaa7920eda8bf9b73da164ae6"} Oct 30 06:19:53.942138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.941917 5565 scope.go:122] "RemoveContainer" containerID="0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91" Oct 30 06:19:53.943966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.943943 5565 generic.go:356] "Generic (PLEG): container finished" podID="4fbb7b13-671d-4383-bef4-2cf446b2e99b" containerID="5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab" exitCode=0 Oct 30 06:19:53.944096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.944080 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" event={"ID":"4fbb7b13-671d-4383-bef4-2cf446b2e99b","Type":"ContainerDied","Data":"5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab"} Oct 30 06:19:53.944161 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.944126 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" event={"ID":"4fbb7b13-671d-4383-bef4-2cf446b2e99b","Type":"ContainerDied","Data":"494b8f715de73c3dbdaafc0c264e401e3f91a06f67d45d0cb7fd3ca4822cdf7b"} Oct 30 06:19:53.944218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.944195 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk" Oct 30 06:19:53.960196 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.960174 5565 scope.go:122] "RemoveContainer" containerID="0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91" Oct 30 06:19:53.960679 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:19:53.960628 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91\": container with ID starting with 0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91 not found: ID does not exist" containerID="0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91" Oct 30 06:19:53.960758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.960692 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91"} err="failed to get container status \"0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91\": rpc error: code = NotFound desc = could not find container \"0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91\": container with ID starting with 0afb6ce66eb7626e547320b4e198616afc8b93e2e3bd3d6529082fcc4a20eb91 not found: ID does not exist" Oct 30 06:19:53.960758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.960717 5565 scope.go:122] "RemoveContainer" containerID="5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab" Oct 30 06:19:53.975180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.974932 5565 scope.go:122] "RemoveContainer" containerID="5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab" Oct 30 06:19:53.975346 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:19:53.975244 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab\": container with ID starting with 5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab not found: ID does not exist" containerID="5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab" Oct 30 06:19:53.975346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:53.975273 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab"} err="failed to get container status \"5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab\": rpc error: code = NotFound desc = could not find container \"5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab\": container with ID starting with 5eb9045e76b84731e769bff3af9517284b0caeafe0bd4ecd0b33c1851e0b3eab not found: ID does not exist" Oct 30 06:19:54.021405 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.021372 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv"] Oct 30 06:19:54.028188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.027807 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-de2bf2-6ffc7d97c5-586fv"] Oct 30 06:19:54.085551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.082198 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk"] Oct 30 06:19:54.093542 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.086419 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-de2bf2-6648b8cd98-zhqqk"] Oct 30 06:19:54.224672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.224644 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m"] Oct 30 06:19:54.225633 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:19:54.225607 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52953f74_1fbf_4c44_be7c_018905c5a213.slice/crio-a1eaab1b2a9d37d784cd55aa144f363eee04ac719cf2adbac2960c0a7185120f WatchSource:0}: Error finding container a1eaab1b2a9d37d784cd55aa144f363eee04ac719cf2adbac2960c0a7185120f: Status 404 returned error can't find the container with id a1eaab1b2a9d37d784cd55aa144f363eee04ac719cf2adbac2960c0a7185120f Oct 30 06:19:54.953163 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.953052 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" event={"ID":"52953f74-1fbf-4c44-be7c-018905c5a213","Type":"ContainerStarted","Data":"fc6c28ebf4632bb7258fc3dcac237f7467fad17bc2cd868d11f250ed29aa66b7"} Oct 30 06:19:54.953643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:54.953163 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" event={"ID":"52953f74-1fbf-4c44-be7c-018905c5a213","Type":"ContainerStarted","Data":"a1eaab1b2a9d37d784cd55aa144f363eee04ac719cf2adbac2960c0a7185120f"} Oct 30 06:19:55.001231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:55.001186 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" podStartSLOduration=2.001173487 podStartE2EDuration="2.001173487s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:19:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:19:55.000423573 +0000 UTC m=+2901.576110648" watchObservedRunningTime="2026-10-30 06:19:55.001173487 +0000 UTC m=+2901.576860564" Oct 30 06:19:55.569426 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:55.569351 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4fbb7b13-671d-4383-bef4-2cf446b2e99b" path="/var/lib/kubelet/pods/4fbb7b13-671d-4383-bef4-2cf446b2e99b/volumes" Oct 30 06:19:55.569809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:55.569793 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70ea4097-fdd9-4811-aa82-1b5d8423de58" path="/var/lib/kubelet/pods/70ea4097-fdd9-4811-aa82-1b5d8423de58/volumes" Oct 30 06:19:56.043888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:56.043843 5565 generic.go:356] "Generic (PLEG): container finished" podID="52953f74-1fbf-4c44-be7c-018905c5a213" containerID="fc6c28ebf4632bb7258fc3dcac237f7467fad17bc2cd868d11f250ed29aa66b7" exitCode=0 Oct 30 06:19:56.044403 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:56.044061 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" event={"ID":"52953f74-1fbf-4c44-be7c-018905c5a213","Type":"ContainerDied","Data":"fc6c28ebf4632bb7258fc3dcac237f7467fad17bc2cd868d11f250ed29aa66b7"} Oct 30 06:19:56.574701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:56.574654 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerName="payload-processing" containerID="cri-o://e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2" gracePeriod=25 Oct 30 06:19:56.609089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:56.609053 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerName="payload-pre-processing" containerID="cri-o://087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6" gracePeriod=25 Oct 30 06:19:57.210042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.210020 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:57.333416 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.333337 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca\") pod \"52953f74-1fbf-4c44-be7c-018905c5a213\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " Oct 30 06:19:57.333594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.333557 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca\") pod \"52953f74-1fbf-4c44-be7c-018905c5a213\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " Oct 30 06:19:57.333723 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.333698 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd\" (UniqueName: \"kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd\") pod \"52953f74-1fbf-4c44-be7c-018905c5a213\" (UID: \"52953f74-1fbf-4c44-be7c-018905c5a213\") " Oct 30 06:19:57.333883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.333861 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca" pod "52953f74-1fbf-4c44-be7c-018905c5a213" (UID: "52953f74-1fbf-4c44-be7c-018905c5a213"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:19:57.334188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.334170 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:57.335536 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.335500 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca" pod "52953f74-1fbf-4c44-be7c-018905c5a213" (UID: "52953f74-1fbf-4c44-be7c-018905c5a213"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:19:57.335791 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.335770 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd" pod "52953f74-1fbf-4c44-be7c-018905c5a213" (UID: "52953f74-1fbf-4c44-be7c-018905c5a213"). InnerVolumeSpecName "kube-api-access-9m9kd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:19:57.435348 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.435316 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m9kd\" (UniqueName: \"kubernetes.io/projected/52953f74-1fbf-4c44-be7c-018905c5a213-kube-api-access-9m9kd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:57.435348 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.435346 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52953f74-1fbf-4c44-be7c-018905c5a213-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:19:57.685628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:57.684378 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:19:58.060217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:58.060182 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" Oct 30 06:19:58.060401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:58.060219 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m" event={"ID":"52953f74-1fbf-4c44-be7c-018905c5a213","Type":"ContainerDied","Data":"a1eaab1b2a9d37d784cd55aa144f363eee04ac719cf2adbac2960c0a7185120f"} Oct 30 06:19:58.060401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:58.060262 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a1eaab1b2a9d37d784cd55aa144f363eee04ac719cf2adbac2960c0a7185120f" Oct 30 06:19:58.665667 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:19:58.665626 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:01.029734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.028321 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" containerID="cri-o://514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55" gracePeriod=45 Oct 30 06:20:01.036813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.036771 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" containerID="cri-o://ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e" gracePeriod=45 Oct 30 06:20:01.146366 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.146333 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn"] Oct 30 06:20:01.147000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.146961 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" containerID="cri-o://05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42" gracePeriod=30 Oct 30 06:20:01.426059 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.304991 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": dial tcp 10.134.0.57:8000: connect: connection refused" Oct 30 06:20:01.538427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.538400 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 06:20:01.547107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547077 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir\") pod \"ca87ec64-c33c-4f05-91e0-e59dda440a82\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " Oct 30 06:20:01.547243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547221 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm\") pod \"ca87ec64-c33c-4f05-91e0-e59dda440a82\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " Oct 30 06:20:01.547303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547271 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs\") pod \"ca87ec64-c33c-4f05-91e0-e59dda440a82\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " Oct 30 06:20:01.547363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547313 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home\") pod \"ca87ec64-c33c-4f05-91e0-e59dda440a82\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " Oct 30 06:20:01.547437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547421 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct\" (UniqueName: \"kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct\") pod \"ca87ec64-c33c-4f05-91e0-e59dda440a82\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " Oct 30 06:20:01.547499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547475 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache\") pod \"ca87ec64-c33c-4f05-91e0-e59dda440a82\" (UID: \"ca87ec64-c33c-4f05-91e0-e59dda440a82\") " Oct 30 06:20:01.547696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547661 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir" pod "ca87ec64-c33c-4f05-91e0-e59dda440a82" (UID: "ca87ec64-c33c-4f05-91e0-e59dda440a82"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.547878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.547854 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home" pod "ca87ec64-c33c-4f05-91e0-e59dda440a82" (UID: "ca87ec64-c33c-4f05-91e0-e59dda440a82"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.548124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.548101 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache" pod "ca87ec64-c33c-4f05-91e0-e59dda440a82" (UID: "ca87ec64-c33c-4f05-91e0-e59dda440a82"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.548309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.548199 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.548429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.548330 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.551480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.551446 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm" pod "ca87ec64-c33c-4f05-91e0-e59dda440a82" (UID: "ca87ec64-c33c-4f05-91e0-e59dda440a82"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.551706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.551685 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs" pod "ca87ec64-c33c-4f05-91e0-e59dda440a82" (UID: "ca87ec64-c33c-4f05-91e0-e59dda440a82"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:01.553255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.553232 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct" pod "ca87ec64-c33c-4f05-91e0-e59dda440a82" (UID: "ca87ec64-c33c-4f05-91e0-e59dda440a82"). InnerVolumeSpecName "kube-api-access-cvnct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:01.568129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.568093 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 06:20:01.649451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.649419 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm\") pod \"f62bec26-ed63-43bf-b0f3-6b73a2464339\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " Oct 30 06:20:01.649614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.649486 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir\") pod \"f62bec26-ed63-43bf-b0f3-6b73a2464339\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " Oct 30 06:20:01.649721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.649688 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq\" (UniqueName: \"kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq\") pod \"f62bec26-ed63-43bf-b0f3-6b73a2464339\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " Oct 30 06:20:01.649789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.649762 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home\") pod \"f62bec26-ed63-43bf-b0f3-6b73a2464339\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " Oct 30 06:20:01.650171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.650149 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs\" (UniqueName: \"kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs\") pod \"f62bec26-ed63-43bf-b0f3-6b73a2464339\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " Oct 30 06:20:01.650469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.650456 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache\") pod \"f62bec26-ed63-43bf-b0f3-6b73a2464339\" (UID: \"f62bec26-ed63-43bf-b0f3-6b73a2464339\") " Oct 30 06:20:01.651658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.651636 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.654135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.654115 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca87ec64-c33c-4f05-91e0-e59dda440a82-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.654354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.654323 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca87ec64-c33c-4f05-91e0-e59dda440a82-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.654499 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.654483 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvnct\" (UniqueName: \"kubernetes.io/projected/ca87ec64-c33c-4f05-91e0-e59dda440a82-kube-api-access-cvnct\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.654631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.654377 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq" pod "f62bec26-ed63-43bf-b0f3-6b73a2464339" (UID: "f62bec26-ed63-43bf-b0f3-6b73a2464339"). InnerVolumeSpecName "kube-api-access-v4lkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:01.666716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.663765 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home" pod "f62bec26-ed63-43bf-b0f3-6b73a2464339" (UID: "f62bec26-ed63-43bf-b0f3-6b73a2464339"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.666716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.666609 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir" pod "f62bec26-ed63-43bf-b0f3-6b73a2464339" (UID: "f62bec26-ed63-43bf-b0f3-6b73a2464339"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.666716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.666656 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache" pod "f62bec26-ed63-43bf-b0f3-6b73a2464339" (UID: "f62bec26-ed63-43bf-b0f3-6b73a2464339"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.667074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.667035 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm" pod "f62bec26-ed63-43bf-b0f3-6b73a2464339" (UID: "f62bec26-ed63-43bf-b0f3-6b73a2464339"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:01.668218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.668182 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs" pod "f62bec26-ed63-43bf-b0f3-6b73a2464339" (UID: "f62bec26-ed63-43bf-b0f3-6b73a2464339"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:01.704477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.704399 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq"] Oct 30 06:20:01.705157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705140 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" Oct 30 06:20:01.705224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705159 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" Oct 30 06:20:01.705224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705193 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" Oct 30 06:20:01.705224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705199 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" Oct 30 06:20:01.705224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705209 5565 cpu_manager.go:414] "containerMap: removing container" podUID="52953f74-1fbf-4c44-be7c-018905c5a213" containerName="revoke-keys" Oct 30 06:20:01.705224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705214 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52953f74-1fbf-4c44-be7c-018905c5a213" containerName="revoke-keys" Oct 30 06:20:01.705224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705223 5565 cpu_manager.go:414] "containerMap: removing container" podUID="70ea4097-fdd9-4811-aa82-1b5d8423de58" containerName="payload-pre-processing" Oct 30 06:20:01.705421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705227 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70ea4097-fdd9-4811-aa82-1b5d8423de58" containerName="payload-pre-processing" Oct 30 06:20:01.705421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705339 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerName="main" Oct 30 06:20:01.705421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705348 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="52953f74-1fbf-4c44-be7c-018905c5a213" containerName="revoke-keys" Oct 30 06:20:01.705421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705357 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="70ea4097-fdd9-4811-aa82-1b5d8423de58" containerName="payload-pre-processing" Oct 30 06:20:01.705421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.705367 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" Oct 30 06:20:01.709521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.709496 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.731632 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.731606 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq"] Oct 30 06:20:01.758607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.758580 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:01.758821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.758744 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:20:01.760491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.760467 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.760655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.760634 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rrtr\" (UniqueName: \"kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.760709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.760698 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.760982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.760961 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.761063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.760987 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f62bec26-ed63-43bf-b0f3-6b73a2464339-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.761063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.761002 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.761063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.761015 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.761063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.761028 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f62bec26-ed63-43bf-b0f3-6b73a2464339-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.761063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.761041 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4lkq\" (UniqueName: \"kubernetes.io/projected/f62bec26-ed63-43bf-b0f3-6b73a2464339-kube-api-access-v4lkq\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:01.862360 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.862325 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.862562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.862403 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rrtr\" (UniqueName: \"kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.862562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.862449 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.863046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.863023 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.864840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.864821 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:01.888037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:01.888012 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rrtr\" (UniqueName: \"kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:02.028522 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.028492 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:02.148634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.148589 5565 generic.go:356] "Generic (PLEG): container finished" podID="f62bec26-ed63-43bf-b0f3-6b73a2464339" containerID="514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55" exitCode=0 Oct 30 06:20:02.149088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.148639 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" Oct 30 06:20:02.149088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.148839 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" event={"ID":"f62bec26-ed63-43bf-b0f3-6b73a2464339","Type":"ContainerDied","Data":"514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55"} Oct 30 06:20:02.149088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.148881 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq" event={"ID":"f62bec26-ed63-43bf-b0f3-6b73a2464339","Type":"ContainerDied","Data":"82a5bc8f4f53a5cfc160372ed589dc81d4e3360835a69fd72ccc7bce9a344810"} Oct 30 06:20:02.149088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.148905 5565 scope.go:122] "RemoveContainer" containerID="514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55" Oct 30 06:20:02.152261 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.152237 5565 generic.go:356] "Generic (PLEG): container finished" podID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerID="ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e" exitCode=0 Oct 30 06:20:02.152492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.152282 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" event={"ID":"ca87ec64-c33c-4f05-91e0-e59dda440a82","Type":"ContainerDied","Data":"ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e"} Oct 30 06:20:02.152643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.152514 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" event={"ID":"ca87ec64-c33c-4f05-91e0-e59dda440a82","Type":"ContainerDied","Data":"51e4f4f732660947ae0147990f6a48a5890014660cc5ea6dd2c40af1e984dbe4"} Oct 30 06:20:02.152643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.152291 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" Oct 30 06:20:02.163060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.163036 5565 scope.go:122] "RemoveContainer" containerID="514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55" Oct 30 06:20:02.163353 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:02.163332 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55\": container with ID starting with 514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55 not found: ID does not exist" containerID="514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55" Oct 30 06:20:02.163420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.163360 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55"} err="failed to get container status \"514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55\": rpc error: code = NotFound desc = could not find container \"514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55\": container with ID starting with 514ae5b01bcb908e1fcbdbd2eb7e7e5cf04ea4d5cb7b813ed0da44480e497d55 not found: ID does not exist" Oct 30 06:20:02.163420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.163381 5565 scope.go:122] "RemoveContainer" containerID="ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e" Oct 30 06:20:02.175330 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.175310 5565 scope.go:122] "RemoveContainer" containerID="ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e" Oct 30 06:20:02.175614 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:02.175596 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e\": container with ID starting with ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e not found: ID does not exist" containerID="ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e" Oct 30 06:20:02.175668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.175625 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e"} err="failed to get container status \"ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e\": rpc error: code = NotFound desc = could not find container \"ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e\": container with ID starting with ff8db897842342fb52ca0d3632b0debdaaf7d4fbef88b4f744b6b012a2eb6b1e not found: ID does not exist" Oct 30 06:20:02.213625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.213602 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq"] Oct 30 06:20:02.219874 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.219848 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/premium-w2-e7a506-kserve-7d779bf4f6-65vwq"] Oct 30 06:20:02.257781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.257761 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n"] Oct 30 06:20:02.265980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.265958 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n"] Oct 30 06:20:02.335661 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.335622 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-e7a506/simulator-w2-e7a506-kserve-5dcbf9d645-wxc9n" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:20:02.336887 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:20:02.336859 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc84fe88_74f2_4174_ab6d_3b17d53805c2.slice/crio-e20f1e38fc8dc825d01218f47ce938de82b658da6cd75bece4a60503b11566f2 WatchSource:0}: Error finding container e20f1e38fc8dc825d01218f47ce938de82b658da6cd75bece4a60503b11566f2: Status 404 returned error can't find the container with id e20f1e38fc8dc825d01218f47ce938de82b658da6cd75bece4a60503b11566f2 Oct 30 06:20:02.336887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:02.336874 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq"] Oct 30 06:20:03.162404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:03.162322 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" event={"ID":"cc84fe88-74f2-4174-ab6d-3b17d53805c2","Type":"ContainerStarted","Data":"ff291cc86bea22f6f545b5ae8aae13b50d2eb937860c655b3891e9e66aa83d4d"} Oct 30 06:20:03.162404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:03.162400 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" event={"ID":"cc84fe88-74f2-4174-ab6d-3b17d53805c2","Type":"ContainerStarted","Data":"e20f1e38fc8dc825d01218f47ce938de82b658da6cd75bece4a60503b11566f2"} Oct 30 06:20:03.209379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:03.209335 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" podStartSLOduration=2.209322552 podStartE2EDuration="2.209322552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:03.206598044 +0000 UTC m=+2909.782285099" watchObservedRunningTime="2026-10-30 06:20:03.209322552 +0000 UTC m=+2909.785009605" Oct 30 06:20:03.571147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:03.571109 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca87ec64-c33c-4f05-91e0-e59dda440a82" path="/var/lib/kubelet/pods/ca87ec64-c33c-4f05-91e0-e59dda440a82/volumes" Oct 30 06:20:03.571799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:03.571774 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f62bec26-ed63-43bf-b0f3-6b73a2464339" path="/var/lib/kubelet/pods/f62bec26-ed63-43bf-b0f3-6b73a2464339/volumes" Oct 30 06:20:04.174073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:04.174038 5565 generic.go:356] "Generic (PLEG): container finished" podID="cc84fe88-74f2-4174-ab6d-3b17d53805c2" containerID="ff291cc86bea22f6f545b5ae8aae13b50d2eb937860c655b3891e9e66aa83d4d" exitCode=0 Oct 30 06:20:04.174493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:04.174108 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" event={"ID":"cc84fe88-74f2-4174-ab6d-3b17d53805c2","Type":"ContainerDied","Data":"ff291cc86bea22f6f545b5ae8aae13b50d2eb937860c655b3891e9e66aa83d4d"} Oct 30 06:20:04.308985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:04.308953 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp"] Oct 30 06:20:04.334102 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:04.334064 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84"] Oct 30 06:20:05.405968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.405944 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:05.530167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.530138 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr\" (UniqueName: \"kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr\") pod \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " Oct 30 06:20:05.530342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.530175 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca\") pod \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " Oct 30 06:20:05.530342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.530232 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca\") pod \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\" (UID: \"cc84fe88-74f2-4174-ab6d-3b17d53805c2\") " Oct 30 06:20:05.530585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.530558 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca" pod "cc84fe88-74f2-4174-ab6d-3b17d53805c2" (UID: "cc84fe88-74f2-4174-ab6d-3b17d53805c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:05.530949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.530932 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:05.532376 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.532358 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr" pod "cc84fe88-74f2-4174-ab6d-3b17d53805c2" (UID: "cc84fe88-74f2-4174-ab6d-3b17d53805c2"). InnerVolumeSpecName "kube-api-access-9rrtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:05.532437 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.532411 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca" pod "cc84fe88-74f2-4174-ab6d-3b17d53805c2" (UID: "cc84fe88-74f2-4174-ab6d-3b17d53805c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:05.632869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.632840 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rrtr\" (UniqueName: \"kubernetes.io/projected/cc84fe88-74f2-4174-ab6d-3b17d53805c2-kube-api-access-9rrtr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:05.632869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:05.632872 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc84fe88-74f2-4174-ab6d-3b17d53805c2-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.236710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.236673 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" event={"ID":"cc84fe88-74f2-4174-ab6d-3b17d53805c2","Type":"ContainerDied","Data":"e20f1e38fc8dc825d01218f47ce938de82b658da6cd75bece4a60503b11566f2"} Oct 30 06:20:06.236840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.236740 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e20f1e38fc8dc825d01218f47ce938de82b658da6cd75bece4a60503b11566f2" Oct 30 06:20:06.236840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.236686 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq" Oct 30 06:20:06.444152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.444127 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:20:06.480297 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.480266 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b"] Oct 30 06:20:06.480640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.480511 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" podUID="9307b9d2-65e1-4c16-952c-817bf260daa4" containerName="maas-api" containerID="cri-o://b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d" gracePeriod=30 Oct 30 06:20:06.544903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.544876 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545097 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545288 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545352 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545405 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545450 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545488 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545556 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.545682 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.545611 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p\") pod \"4cb2b128-3deb-4ef7-b632-019fac593c36\" (UID: \"4cb2b128-3deb-4ef7-b632-019fac593c36\") " Oct 30 06:20:06.546095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.546068 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:06.546247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.546227 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4cb2b128-3deb-4ef7-b632-019fac593c36-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.546785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.546725 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.546954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.546932 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.547089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.547020 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.547241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.547216 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.549657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.549629 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:06.553023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.552996 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:06.554762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.554718 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.555514 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.555492 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p" pod "4cb2b128-3deb-4ef7-b632-019fac593c36" (UID: "4cb2b128-3deb-4ef7-b632-019fac593c36"). InnerVolumeSpecName "kube-api-access-zfz6p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648541 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648575 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648588 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648604 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648618 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfz6p\" (UniqueName: \"kubernetes.io/projected/4cb2b128-3deb-4ef7-b632-019fac593c36-kube-api-access-zfz6p\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648636 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648649 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.648690 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.648662 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4cb2b128-3deb-4ef7-b632-019fac593c36-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.660496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.660458 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd"] Oct 30 06:20:06.665800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.665743 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" podUID="b4a2fd88-6c33-4ebf-afba-0f8a87720e92" containerName="payload-processing" containerID="cri-o://306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347" gracePeriod=30 Oct 30 06:20:06.678020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.673279 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl"] Oct 30 06:20:06.678020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.673582 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" podUID="403ff440-ad88-4ce8-95b4-1aa3f07cb389" containerName="payload-pre-processing" containerID="cri-o://dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784" gracePeriod=30 Oct 30 06:20:06.845397 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.844776 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:20:06.954363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.954329 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6\" (UniqueName: \"kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6\") pod \"9307b9d2-65e1-4c16-952c-817bf260daa4\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " Oct 30 06:20:06.955103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.955037 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs\") pod \"9307b9d2-65e1-4c16-952c-817bf260daa4\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " Oct 30 06:20:06.955383 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.955360 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca\") pod \"9307b9d2-65e1-4c16-952c-817bf260daa4\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " Oct 30 06:20:06.955475 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.955408 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls\") pod \"9307b9d2-65e1-4c16-952c-817bf260daa4\" (UID: \"9307b9d2-65e1-4c16-952c-817bf260daa4\") " Oct 30 06:20:06.955810 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.955783 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca" pod "9307b9d2-65e1-4c16-952c-817bf260daa4" (UID: "9307b9d2-65e1-4c16-952c-817bf260daa4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:06.956189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.956166 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9307b9d2-65e1-4c16-952c-817bf260daa4-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:06.958405 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.958367 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6" pod "9307b9d2-65e1-4c16-952c-817bf260daa4" (UID: "9307b9d2-65e1-4c16-952c-817bf260daa4"). InnerVolumeSpecName "kube-api-access-c96b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:06.958784 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.958742 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls" pod "9307b9d2-65e1-4c16-952c-817bf260daa4" (UID: "9307b9d2-65e1-4c16-952c-817bf260daa4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:06.960599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.960565 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs" pod "9307b9d2-65e1-4c16-952c-817bf260daa4" (UID: "9307b9d2-65e1-4c16-952c-817bf260daa4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:20:06.978576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:06.978549 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:20:07.121189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.121149 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume\") pod \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " Oct 30 06:20:07.121382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.121299 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs\" (UniqueName: \"kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs\") pod \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\" (UID: \"403ff440-ad88-4ce8-95b4-1aa3f07cb389\") " Oct 30 06:20:07.121967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.121941 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.122111 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.121970 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c96b6\" (UniqueName: \"kubernetes.io/projected/9307b9d2-65e1-4c16-952c-817bf260daa4-kube-api-access-c96b6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.122111 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.121986 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9307b9d2-65e1-4c16-952c-817bf260daa4-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.144450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.144391 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume" pod "403ff440-ad88-4ce8-95b4-1aa3f07cb389" (UID: "403ff440-ad88-4ce8-95b4-1aa3f07cb389"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:07.161585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.150013 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs" pod "403ff440-ad88-4ce8-95b4-1aa3f07cb389" (UID: "403ff440-ad88-4ce8-95b4-1aa3f07cb389"). InnerVolumeSpecName "kube-api-access-5njqs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:07.180054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.179703 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:20:07.223361 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.223269 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb\" (UniqueName: \"kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb\") pod \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " Oct 30 06:20:07.223556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.223420 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume\") pod \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\" (UID: \"b4a2fd88-6c33-4ebf-afba-0f8a87720e92\") " Oct 30 06:20:07.224094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.224062 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume" pod "b4a2fd88-6c33-4ebf-afba-0f8a87720e92" (UID: "b4a2fd88-6c33-4ebf-afba-0f8a87720e92"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:07.224094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.224087 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5njqs\" (UniqueName: \"kubernetes.io/projected/403ff440-ad88-4ce8-95b4-1aa3f07cb389-kube-api-access-5njqs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.224257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.224112 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/403ff440-ad88-4ce8-95b4-1aa3f07cb389-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.233254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.233051 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb" pod "b4a2fd88-6c33-4ebf-afba-0f8a87720e92" (UID: "b4a2fd88-6c33-4ebf-afba-0f8a87720e92"). InnerVolumeSpecName "kube-api-access-hdhkb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:07.252031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.252000 5565 generic.go:356] "Generic (PLEG): container finished" podID="b4a2fd88-6c33-4ebf-afba-0f8a87720e92" containerID="306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347" exitCode=0 Oct 30 06:20:07.252351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.252323 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" event={"ID":"b4a2fd88-6c33-4ebf-afba-0f8a87720e92","Type":"ContainerDied","Data":"306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347"} Oct 30 06:20:07.252515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.252501 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" event={"ID":"b4a2fd88-6c33-4ebf-afba-0f8a87720e92","Type":"ContainerDied","Data":"1dcc539138dfe996953023a8396496e28acedf6123afae3053ab67d2d0e407be"} Oct 30 06:20:07.253010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.252628 5565 scope.go:122] "RemoveContainer" containerID="306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347" Oct 30 06:20:07.253245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.252679 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd" Oct 30 06:20:07.255461 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.255430 5565 generic.go:356] "Generic (PLEG): container finished" podID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerID="05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42" exitCode=0 Oct 30 06:20:07.255588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.255541 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" event={"ID":"4cb2b128-3deb-4ef7-b632-019fac593c36","Type":"ContainerDied","Data":"05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42"} Oct 30 06:20:07.255649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.255585 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" event={"ID":"4cb2b128-3deb-4ef7-b632-019fac593c36","Type":"ContainerDied","Data":"d7288a3267762068b1dbb85032a9fcb2f1a649ea5a3bfc69b0c142e796ca7f73"} Oct 30 06:20:07.255738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.255678 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn" Oct 30 06:20:07.262573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.260397 5565 generic.go:356] "Generic (PLEG): container finished" podID="403ff440-ad88-4ce8-95b4-1aa3f07cb389" containerID="dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784" exitCode=0 Oct 30 06:20:07.262573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.260472 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" Oct 30 06:20:07.262573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.260512 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" event={"ID":"403ff440-ad88-4ce8-95b4-1aa3f07cb389","Type":"ContainerDied","Data":"dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784"} Oct 30 06:20:07.262573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.260567 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl" event={"ID":"403ff440-ad88-4ce8-95b4-1aa3f07cb389","Type":"ContainerDied","Data":"ed9460421aa0251168132852dbb8883d0517afb5bf5c5c37a114437c4360e226"} Oct 30 06:20:07.272127 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.272100 5565 generic.go:356] "Generic (PLEG): container finished" podID="9307b9d2-65e1-4c16-952c-817bf260daa4" containerID="b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d" exitCode=0 Oct 30 06:20:07.272247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.272149 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" Oct 30 06:20:07.272247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.272176 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" event={"ID":"9307b9d2-65e1-4c16-952c-817bf260daa4","Type":"ContainerDied","Data":"b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d"} Oct 30 06:20:07.272247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.272213 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b" event={"ID":"9307b9d2-65e1-4c16-952c-817bf260daa4","Type":"ContainerDied","Data":"2cf108339781aba4d77ebab15a6d048c1e3ba6a731f9dbf854f41f792f2eaf38"} Oct 30 06:20:07.285433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.285253 5565 scope.go:122] "RemoveContainer" containerID="306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347" Oct 30 06:20:07.285592 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:07.285573 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347\": container with ID starting with 306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347 not found: ID does not exist" containerID="306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347" Oct 30 06:20:07.285681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.285602 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347"} err="failed to get container status \"306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347\": rpc error: code = NotFound desc = could not find container \"306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347\": container with ID starting with 306c5f2f269a27c0d43e69e2da61eb07c497d1cb7f9257e676ba35820ffb7347 not found: ID does not exist" Oct 30 06:20:07.285681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.285625 5565 scope.go:122] "RemoveContainer" containerID="05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42" Oct 30 06:20:07.297157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.297136 5565 scope.go:122] "RemoveContainer" containerID="05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42" Oct 30 06:20:07.297431 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:07.297410 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42\": container with ID starting with 05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42 not found: ID does not exist" containerID="05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42" Oct 30 06:20:07.297517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.297441 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42"} err="failed to get container status \"05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42\": rpc error: code = NotFound desc = could not find container \"05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42\": container with ID starting with 05ef7c1f876753c862896271dd388706cd8aa735bbf57ea183fd8512ed0e5c42 not found: ID does not exist" Oct 30 06:20:07.297517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.297464 5565 scope.go:122] "RemoveContainer" containerID="dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784" Oct 30 06:20:07.307298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.307278 5565 scope.go:122] "RemoveContainer" containerID="dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784" Oct 30 06:20:07.307581 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:07.307553 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784\": container with ID starting with dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784 not found: ID does not exist" containerID="dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784" Oct 30 06:20:07.307666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.307589 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784"} err="failed to get container status \"dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784\": rpc error: code = NotFound desc = could not find container \"dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784\": container with ID starting with dec668666a57905496211790a66e850ee2fe672ffbc23d4b835c0b45b05e8784 not found: ID does not exist" Oct 30 06:20:07.307666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.307612 5565 scope.go:122] "RemoveContainer" containerID="b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d" Oct 30 06:20:07.316773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.316755 5565 scope.go:122] "RemoveContainer" containerID="b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d" Oct 30 06:20:07.317032 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:07.317007 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d\": container with ID starting with b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d not found: ID does not exist" containerID="b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d" Oct 30 06:20:07.317105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.317038 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d"} err="failed to get container status \"b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d\": rpc error: code = NotFound desc = could not find container \"b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d\": container with ID starting with b4476b445b048a7b0c243ae3c57221c00b823d3b36641c5ea8ef126fcd92b53d not found: ID does not exist" Oct 30 06:20:07.325392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.325371 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdhkb\" (UniqueName: \"kubernetes.io/projected/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-kube-api-access-hdhkb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.325392 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.325393 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b4a2fd88-6c33-4ebf-afba-0f8a87720e92-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:07.331987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.331961 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b"] Oct 30 06:20:07.337864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.337843 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-36ae27-77fb86d7df-rjs4b"] Oct 30 06:20:07.376036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.376013 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn"] Oct 30 06:20:07.380291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.380270 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-de2bf2-openshift-default-84d9654c67-bx6vn"] Oct 30 06:20:07.418194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.418174 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl"] Oct 30 06:20:07.428554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.428518 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-36ae27-7559497855-p4jvl"] Oct 30 06:20:07.433278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.433259 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7"] Oct 30 06:20:07.433955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.433940 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" Oct 30 06:20:07.434020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.433957 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" Oct 30 06:20:07.434020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.433973 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cc84fe88-74f2-4174-ab6d-3b17d53805c2" containerName="revoke-keys" Oct 30 06:20:07.434020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.433979 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc84fe88-74f2-4174-ab6d-3b17d53805c2" containerName="revoke-keys" Oct 30 06:20:07.434020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434017 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9307b9d2-65e1-4c16-952c-817bf260daa4" containerName="maas-api" Oct 30 06:20:07.434020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434021 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9307b9d2-65e1-4c16-952c-817bf260daa4" containerName="maas-api" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434036 5565 cpu_manager.go:414] "containerMap: removing container" podUID="403ff440-ad88-4ce8-95b4-1aa3f07cb389" containerName="payload-pre-processing" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434041 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="403ff440-ad88-4ce8-95b4-1aa3f07cb389" containerName="payload-pre-processing" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434050 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b4a2fd88-6c33-4ebf-afba-0f8a87720e92" containerName="payload-processing" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434055 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a2fd88-6c33-4ebf-afba-0f8a87720e92" containerName="payload-processing" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434154 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a2fd88-6c33-4ebf-afba-0f8a87720e92" containerName="payload-processing" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434165 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" containerName="istio-proxy" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434173 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc84fe88-74f2-4174-ab6d-3b17d53805c2" containerName="revoke-keys" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434187 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9307b9d2-65e1-4c16-952c-817bf260daa4" containerName="maas-api" Oct 30 06:20:07.434215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.434195 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="403ff440-ad88-4ce8-95b4-1aa3f07cb389" containerName="payload-pre-processing" Oct 30 06:20:07.438971 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.438955 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.456540 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.456505 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7"] Oct 30 06:20:07.457672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.457656 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-d4bace-serving-cert\"" Oct 30 06:20:07.473075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.473047 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd"] Oct 30 06:20:07.476686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.476627 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-36ae27-7f77fb499d-c88jd"] Oct 30 06:20:07.528182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.528151 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.528302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.528220 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdkjg\" (UniqueName: \"kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.528367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.528353 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.528411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.528399 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.570628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.570591 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="403ff440-ad88-4ce8-95b4-1aa3f07cb389" path="/var/lib/kubelet/pods/403ff440-ad88-4ce8-95b4-1aa3f07cb389/volumes" Oct 30 06:20:07.571202 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.571179 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cb2b128-3deb-4ef7-b632-019fac593c36" path="/var/lib/kubelet/pods/4cb2b128-3deb-4ef7-b632-019fac593c36/volumes" Oct 30 06:20:07.571725 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.571701 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9307b9d2-65e1-4c16-952c-817bf260daa4" path="/var/lib/kubelet/pods/9307b9d2-65e1-4c16-952c-817bf260daa4/volumes" Oct 30 06:20:07.572520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.572505 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a2fd88-6c33-4ebf-afba-0f8a87720e92" path="/var/lib/kubelet/pods/b4a2fd88-6c33-4ebf-afba-0f8a87720e92/volumes" Oct 30 06:20:07.629728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.629703 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.629865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.629799 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.629865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.629844 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdkjg\" (UniqueName: \"kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.629992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.629959 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.630367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.630345 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.632227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.632209 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.632345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.632328 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.654585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.654513 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdkjg\" (UniqueName: \"kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg\") pod \"maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:07.659340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.659310 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:07.754457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:07.754431 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:08.063861 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:08.063827 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7"] Oct 30 06:20:08.064604 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:20:08.064574 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14da1d18_aa12_4d44_8a55_5217833d0686.slice/crio-6fffeaaa7f4245bf5fdcd48bfc7c9e43e838d9cb41833b8b03cde0637534a066 WatchSource:0}: Error finding container 6fffeaaa7f4245bf5fdcd48bfc7c9e43e838d9cb41833b8b03cde0637534a066: Status 404 returned error can't find the container with id 6fffeaaa7f4245bf5fdcd48bfc7c9e43e838d9cb41833b8b03cde0637534a066 Oct 30 06:20:08.066072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:08.066048 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:20:08.281566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:08.281521 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" event={"ID":"14da1d18-aa12-4d44-8a55-5217833d0686","Type":"ContainerStarted","Data":"6fffeaaa7f4245bf5fdcd48bfc7c9e43e838d9cb41833b8b03cde0637534a066"} Oct 30 06:20:08.664339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:08.664243 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:20:09.294132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.294096 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" event={"ID":"14da1d18-aa12-4d44-8a55-5217833d0686","Type":"ContainerStarted","Data":"a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106"} Oct 30 06:20:09.294397 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.294267 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:09.309745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.309717 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" podUID="cd0eaa1e-aca6-4b5d-abe2-a55342f64702" containerName="payload-processing" containerID="cri-o://4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49" gracePeriod=25 Oct 30 06:20:09.335599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.335574 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" podUID="cd677b6c-37b8-44b3-8e70-3d25048be781" containerName="payload-pre-processing" containerID="cri-o://177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f" gracePeriod=25 Oct 30 06:20:09.340994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.340953 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" podStartSLOduration=2.006991531 podStartE2EDuration="2.340938125s" totalImagesPullingTime="333.946594ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:09.339497664 +0000 UTC m=+2915.915184718" watchObservedRunningTime="2026-10-30 06:20:09.340938125 +0000 UTC m=+2915.916625181" Oct 30 06:20:09.661959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.661935 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:20:09.688359 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.688331 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:20:09.695049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695025 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume\") pod \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " Oct 30 06:20:09.695153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695072 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume\") pod \"cd677b6c-37b8-44b3-8e70-3d25048be781\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " Oct 30 06:20:09.695153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695103 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff\" (UniqueName: \"kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff\") pod \"cd677b6c-37b8-44b3-8e70-3d25048be781\" (UID: \"cd677b6c-37b8-44b3-8e70-3d25048be781\") " Oct 30 06:20:09.695262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695220 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2\" (UniqueName: \"kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2\") pod \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\" (UID: \"cd0eaa1e-aca6-4b5d-abe2-a55342f64702\") " Oct 30 06:20:09.695444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695419 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume" pod "cd0eaa1e-aca6-4b5d-abe2-a55342f64702" (UID: "cd0eaa1e-aca6-4b5d-abe2-a55342f64702"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:09.695444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695431 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume" pod "cd677b6c-37b8-44b3-8e70-3d25048be781" (UID: "cd677b6c-37b8-44b3-8e70-3d25048be781"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:09.695847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695826 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:09.695916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.695851 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd677b6c-37b8-44b3-8e70-3d25048be781-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:09.697075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.697056 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff" pod "cd677b6c-37b8-44b3-8e70-3d25048be781" (UID: "cd677b6c-37b8-44b3-8e70-3d25048be781"). InnerVolumeSpecName "kube-api-access-kr6ff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:09.697209 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.697186 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2" pod "cd0eaa1e-aca6-4b5d-abe2-a55342f64702" (UID: "cd0eaa1e-aca6-4b5d-abe2-a55342f64702"). InnerVolumeSpecName "kube-api-access-ggnt2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:09.806124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.806076 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr6ff\" (UniqueName: \"kubernetes.io/projected/cd677b6c-37b8-44b3-8e70-3d25048be781-kube-api-access-kr6ff\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:09.806124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.806119 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ggnt2\" (UniqueName: \"kubernetes.io/projected/cd0eaa1e-aca6-4b5d-abe2-a55342f64702-kube-api-access-ggnt2\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:09.956210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.956133 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl"] Oct 30 06:20:09.956892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.956875 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cd677b6c-37b8-44b3-8e70-3d25048be781" containerName="payload-pre-processing" Oct 30 06:20:09.956952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.956893 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd677b6c-37b8-44b3-8e70-3d25048be781" containerName="payload-pre-processing" Oct 30 06:20:09.956952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.956920 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cd0eaa1e-aca6-4b5d-abe2-a55342f64702" containerName="payload-processing" Oct 30 06:20:09.956952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.956925 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd0eaa1e-aca6-4b5d-abe2-a55342f64702" containerName="payload-processing" Oct 30 06:20:09.957077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.957030 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd677b6c-37b8-44b3-8e70-3d25048be781" containerName="payload-pre-processing" Oct 30 06:20:09.957077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.957041 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd0eaa1e-aca6-4b5d-abe2-a55342f64702" containerName="payload-processing" Oct 30 06:20:09.960728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.960711 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:09.979523 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.979496 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-718730\"" Oct 30 06:20:09.980465 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:09.980440 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl"] Oct 30 06:20:10.008689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.008659 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhvbn\" (UniqueName: \"kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn\") pod \"payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.008826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.008728 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.131553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.123600 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.131553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.123898 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhvbn\" (UniqueName: \"kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn\") pod \"payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.131553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.125173 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.176551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.175605 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhvbn\" (UniqueName: \"kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn\") pod \"payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.274337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.274298 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:10.304980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.304946 5565 generic.go:356] "Generic (PLEG): container finished" podID="cd677b6c-37b8-44b3-8e70-3d25048be781" containerID="177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f" exitCode=0 Oct 30 06:20:10.305203 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.305177 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" event={"ID":"cd677b6c-37b8-44b3-8e70-3d25048be781","Type":"ContainerDied","Data":"177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f"} Oct 30 06:20:10.305295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.305242 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" event={"ID":"cd677b6c-37b8-44b3-8e70-3d25048be781","Type":"ContainerDied","Data":"f9a60cde443bcb696d6043f34efe087664a825db5b4796bc54b47607aa89c17c"} Oct 30 06:20:10.305295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.305275 5565 scope.go:122] "RemoveContainer" containerID="177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f" Oct 30 06:20:10.305447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.305430 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84" Oct 30 06:20:10.315254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.315118 5565 generic.go:356] "Generic (PLEG): container finished" podID="cd0eaa1e-aca6-4b5d-abe2-a55342f64702" containerID="4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49" exitCode=0 Oct 30 06:20:10.315254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.315227 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" Oct 30 06:20:10.316834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.316695 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" event={"ID":"cd0eaa1e-aca6-4b5d-abe2-a55342f64702","Type":"ContainerDied","Data":"4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49"} Oct 30 06:20:10.316834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.316753 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp" event={"ID":"cd0eaa1e-aca6-4b5d-abe2-a55342f64702","Type":"ContainerDied","Data":"2ef0bf8f143dd84a18cebc3e1d24c453f37bc10fb6bb7db200438f7604697fa6"} Oct 30 06:20:10.325664 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.325369 5565 scope.go:122] "RemoveContainer" containerID="177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f" Oct 30 06:20:10.325734 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:10.325658 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f\": container with ID starting with 177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f not found: ID does not exist" containerID="177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f" Oct 30 06:20:10.325734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.325681 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f"} err="failed to get container status \"177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f\": rpc error: code = NotFound desc = could not find container \"177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f\": container with ID starting with 177313b830bb6d94a21bd8f850d9e3c83a648662df3dc4c6ddcf649bc8e7724f not found: ID does not exist" Oct 30 06:20:10.325734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.325698 5565 scope.go:122] "RemoveContainer" containerID="4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49" Oct 30 06:20:10.341658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.341639 5565 scope.go:122] "RemoveContainer" containerID="4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49" Oct 30 06:20:10.341989 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:10.341964 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49\": container with ID starting with 4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49 not found: ID does not exist" containerID="4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49" Oct 30 06:20:10.342083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.341997 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49"} err="failed to get container status \"4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49\": rpc error: code = NotFound desc = could not find container \"4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49\": container with ID starting with 4528ef6dbed9c4212ab0b87e6a7293dd9a94a1666d26a3b388cc61346d88ef49 not found: ID does not exist" Oct 30 06:20:10.365824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.365803 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84"] Oct 30 06:20:10.376070 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.376041 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-6795dfcd5f-9rr84"] Oct 30 06:20:10.429629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.429593 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp"] Oct 30 06:20:10.434862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.434827 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a8b892-78746f8f98-v4tzp"] Oct 30 06:20:10.600482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:10.600458 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl"] Oct 30 06:20:10.600666 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:20:10.600647 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd6f0329_e9ac_4d7f_8e97_1a1b8b5b196d.slice/crio-780df681229c5bc30cf0194bc0deb99d4a71517359246d25dc778fb754fcbead WatchSource:0}: Error finding container 780df681229c5bc30cf0194bc0deb99d4a71517359246d25dc778fb754fcbead: Status 404 returned error can't find the container with id 780df681229c5bc30cf0194bc0deb99d4a71517359246d25dc778fb754fcbead Oct 30 06:20:11.327648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:11.327611 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" event={"ID":"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d","Type":"ContainerStarted","Data":"8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7"} Oct 30 06:20:11.328256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:11.327672 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" event={"ID":"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d","Type":"ContainerStarted","Data":"780df681229c5bc30cf0194bc0deb99d4a71517359246d25dc778fb754fcbead"} Oct 30 06:20:11.328256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:11.327699 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:11.401292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:11.397325 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" podStartSLOduration=2.397304706 podStartE2EDuration="2.397304706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:11.387203149 +0000 UTC m=+2917.962890371" watchObservedRunningTime="2026-10-30 06:20:11.397304706 +0000 UTC m=+2917.972991761" Oct 30 06:20:11.570292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:11.570258 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd0eaa1e-aca6-4b5d-abe2-a55342f64702" path="/var/lib/kubelet/pods/cd0eaa1e-aca6-4b5d-abe2-a55342f64702/volumes" Oct 30 06:20:11.570821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:11.570803 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd677b6c-37b8-44b3-8e70-3d25048be781" path="/var/lib/kubelet/pods/cd677b6c-37b8-44b3-8e70-3d25048be781/volumes" Oct 30 06:20:15.323871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:15.323842 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:20:16.165536 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.165491 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:20:16.898813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.898782 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 06:20:16.919076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.919054 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 06:20:16.994041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.994014 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl\" (UniqueName: \"kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl\") pod \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " Oct 30 06:20:16.994235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.994095 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv\" (UniqueName: \"kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv\") pod \"715c1824-5f59-4aec-9191-a6abe7f3085b\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " Oct 30 06:20:16.994314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.994237 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume\") pod \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\" (UID: \"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c\") " Oct 30 06:20:16.994382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.994318 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume\") pod \"715c1824-5f59-4aec-9191-a6abe7f3085b\" (UID: \"715c1824-5f59-4aec-9191-a6abe7f3085b\") " Oct 30 06:20:16.994701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.994674 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume" pod "715c1824-5f59-4aec-9191-a6abe7f3085b" (UID: "715c1824-5f59-4aec-9191-a6abe7f3085b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:16.994817 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.994674 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume" pod "9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" (UID: "9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:16.997947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.995169 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:16.997947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.995195 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/715c1824-5f59-4aec-9191-a6abe7f3085b-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:16.997947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.996467 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv" pod "715c1824-5f59-4aec-9191-a6abe7f3085b" (UID: "715c1824-5f59-4aec-9191-a6abe7f3085b"). InnerVolumeSpecName "kube-api-access-2mvkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:16.997947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:16.996752 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl" pod "9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" (UID: "9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c"). InnerVolumeSpecName "kube-api-access-zbwkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:17.096379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.096349 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zbwkl\" (UniqueName: \"kubernetes.io/projected/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c-kube-api-access-zbwkl\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:17.096379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.096380 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mvkv\" (UniqueName: \"kubernetes.io/projected/715c1824-5f59-4aec-9191-a6abe7f3085b-kube-api-access-2mvkv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:17.422111 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.422065 5565 generic.go:356] "Generic (PLEG): container finished" podID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerID="e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2" exitCode=0 Oct 30 06:20:17.422332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.422220 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" event={"ID":"715c1824-5f59-4aec-9191-a6abe7f3085b","Type":"ContainerDied","Data":"e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2"} Oct 30 06:20:17.422332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.422305 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" event={"ID":"715c1824-5f59-4aec-9191-a6abe7f3085b","Type":"ContainerDied","Data":"dbcb6e7d4e256e5e4bdd03be4d9d6b5b5eab35ca612f2624bc7a33a0b41975c4"} Oct 30 06:20:17.422472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.422346 5565 scope.go:122] "RemoveContainer" containerID="e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2" Oct 30 06:20:17.423006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.422982 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v" Oct 30 06:20:17.425028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.424997 5565 generic.go:356] "Generic (PLEG): container finished" podID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerID="087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6" exitCode=0 Oct 30 06:20:17.425153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.425133 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" event={"ID":"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c","Type":"ContainerDied","Data":"087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6"} Oct 30 06:20:17.425219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.425177 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" event={"ID":"9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c","Type":"ContainerDied","Data":"cdf077dd09848e83f4e341c392404259fd886b6260bc4775c9b55f342a16a4fe"} Oct 30 06:20:17.425297 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.425278 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b" Oct 30 06:20:17.441782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.441762 5565 scope.go:122] "RemoveContainer" containerID="e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2" Oct 30 06:20:17.442435 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:17.442062 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2\": container with ID starting with e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2 not found: ID does not exist" containerID="e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2" Oct 30 06:20:17.442435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.442099 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2"} err="failed to get container status \"e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2\": rpc error: code = NotFound desc = could not find container \"e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2\": container with ID starting with e6f6b3b2e6ae3b928293f85723e719fd2d1db8f011d956457e92e71e594a8df2 not found: ID does not exist" Oct 30 06:20:17.442435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.442118 5565 scope.go:122] "RemoveContainer" containerID="087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6" Oct 30 06:20:17.481942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.481907 5565 scope.go:122] "RemoveContainer" containerID="087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6" Oct 30 06:20:17.482242 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:17.482218 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6\": container with ID starting with 087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6 not found: ID does not exist" containerID="087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6" Oct 30 06:20:17.482328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.482253 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6"} err="failed to get container status \"087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6\": rpc error: code = NotFound desc = could not find container \"087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6\": container with ID starting with 087a261ebfc308521c395e6af0c3d58f0e2051630c810394ac2f196d37b308f6 not found: ID does not exist" Oct 30 06:20:17.499363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.499331 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v"] Oct 30 06:20:17.513276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.513253 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-e7a506-7bb6f57486-pl55v"] Oct 30 06:20:17.533332 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:17.533303 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:20:17.551693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.551666 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b"] Oct 30 06:20:17.554941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.554921 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-e7a506-557bf7dbf5-zwm7b"] Oct 30 06:20:17.570489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.570466 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" path="/var/lib/kubelet/pods/715c1824-5f59-4aec-9191-a6abe7f3085b/volumes" Oct 30 06:20:17.570946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:17.570928 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" path="/var/lib/kubelet/pods/9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c/volumes" Oct 30 06:20:22.360945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:22.360917 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:20:35.787639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:35.787601 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67"] Oct 30 06:20:35.801272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:35.801238 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-e7a506-4ln67"] Oct 30 06:20:37.568377 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:37.568338 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dea1b9f7-4502-4c29-b6ac-41309e2a5113" path="/var/lib/kubelet/pods/dea1b9f7-4502-4c29-b6ac-41309e2a5113/volumes" Oct 30 06:20:42.874512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:42.874475 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2"] Oct 30 06:20:42.875135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:42.874810 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" containerID="cri-o://b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100" gracePeriod=30 Oct 30 06:20:44.278557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.270246 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br"] Oct 30 06:20:44.282750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.282722 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerName="payload-pre-processing" Oct 30 06:20:44.282986 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.282967 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerName="payload-pre-processing" Oct 30 06:20:44.283204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.283192 5565 cpu_manager.go:414] "containerMap: removing container" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerName="payload-processing" Oct 30 06:20:44.283308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.283296 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerName="payload-processing" Oct 30 06:20:44.283708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.283673 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="715c1824-5f59-4aec-9191-a6abe7f3085b" containerName="payload-processing" Oct 30 06:20:44.283879 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.283865 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f73b1dc-1476-4eb4-80fb-ab0f38b8dc7c" containerName="payload-pre-processing" Oct 30 06:20:44.309738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.309708 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br"] Oct 30 06:20:44.310099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.310025 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.342916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.342872 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a8b892\"" Oct 30 06:20:44.344546 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.344499 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a8b892-dockercfg-pfnrr\"" Oct 30 06:20:44.374787 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.374752 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.375160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.375119 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.476498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.476459 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.476701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.476593 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.477255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.477230 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.507642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.507613 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr\") pod \"payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.630794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.630725 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:44.942216 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:44.942186 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br"] Oct 30 06:20:44.942766 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:20:44.942736 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0a8f40_9241_49f3_b904_7e0df242f7f8.slice/crio-13365c176aac6c2bc394a68a5213b080b41132f0eb2f52fa1013175c32f6a95a WatchSource:0}: Error finding container 13365c176aac6c2bc394a68a5213b080b41132f0eb2f52fa1013175c32f6a95a: Status 404 returned error can't find the container with id 13365c176aac6c2bc394a68a5213b080b41132f0eb2f52fa1013175c32f6a95a Oct 30 06:20:45.700966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:45.700926 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" event={"ID":"9d0a8f40-9241-49f3-b904-7e0df242f7f8","Type":"ContainerStarted","Data":"434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565"} Oct 30 06:20:45.701360 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:45.700986 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" event={"ID":"9d0a8f40-9241-49f3-b904-7e0df242f7f8","Type":"ContainerStarted","Data":"13365c176aac6c2bc394a68a5213b080b41132f0eb2f52fa1013175c32f6a95a"} Oct 30 06:20:45.701360 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:45.701008 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:45.745350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:45.745282 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" podStartSLOduration=1.74526372 podStartE2EDuration="1.74526372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:45.744939697 +0000 UTC m=+2952.320626752" watchObservedRunningTime="2026-10-30 06:20:45.74526372 +0000 UTC m=+2952.320950775" Oct 30 06:20:47.534441 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:47.534400 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:20:48.242298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.241912 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 06:20:48.315698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.311983 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.315990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.315964 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.316184 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.316168 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.316343 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.316327 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.316562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.316544 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.316773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.316756 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.316899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.316885 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.317091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.317077 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.317303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.317290 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token\") pod \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\" (UID: \"99a0f929-f527-4920-ac51-6c0fbbfea9a9\") " Oct 30 06:20:48.320275 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.320246 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.320869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.320560 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:20:48.320869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.320803 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.321314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.321097 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.322444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.321470 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.355553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.352352 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:20:48.422142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.422007 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424774 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424806 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424821 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424835 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424848 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424861 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.425018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.424874 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99a0f929-f527-4920-ac51-6c0fbbfea9a9-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.431234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.428161 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:48.431234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.430672 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs" pod "99a0f929-f527-4920-ac51-6c0fbbfea9a9" (UID: "99a0f929-f527-4920-ac51-6c0fbbfea9a9"). InnerVolumeSpecName "kube-api-access-rvjjs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:20:48.526030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.525995 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.526030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.526029 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvjjs\" (UniqueName: \"kubernetes.io/projected/99a0f929-f527-4920-ac51-6c0fbbfea9a9-kube-api-access-rvjjs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:20:48.725335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.725299 5565 generic.go:356] "Generic (PLEG): container finished" podID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerID="b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100" exitCode=0 Oct 30 06:20:48.725825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.725344 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" Oct 30 06:20:48.725825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.725391 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" event={"ID":"99a0f929-f527-4920-ac51-6c0fbbfea9a9","Type":"ContainerDied","Data":"b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100"} Oct 30 06:20:48.725825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.725455 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2" event={"ID":"99a0f929-f527-4920-ac51-6c0fbbfea9a9","Type":"ContainerDied","Data":"11edf119783cbc9dfa0dd755e1cfa11e1a9ef0d9d60cdda351fe4aaf3f175ce0"} Oct 30 06:20:48.725825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.725493 5565 scope.go:122] "RemoveContainer" containerID="b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100" Oct 30 06:20:48.746170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.746147 5565 scope.go:122] "RemoveContainer" containerID="b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100" Oct 30 06:20:48.746499 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:20:48.746477 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100\": container with ID starting with b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100 not found: ID does not exist" containerID="b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100" Oct 30 06:20:48.746620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.746508 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100"} err="failed to get container status \"b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100\": rpc error: code = NotFound desc = could not find container \"b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100\": container with ID starting with b0e6b635ef0494ccab1242ffaa31651983c6903f9f2bf997e342b513d3e75100 not found: ID does not exist" Oct 30 06:20:48.785683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.785619 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2"] Oct 30 06:20:48.799356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.799325 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-e7a506-openshift-default-6bc747d5b9-npnq2"] Oct 30 06:20:48.941939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.941909 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq"] Oct 30 06:20:48.947946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:48.947922 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-1b4b7a-tb8rq"] Oct 30 06:20:49.568466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:49.568433 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" path="/var/lib/kubelet/pods/99a0f929-f527-4920-ac51-6c0fbbfea9a9/volumes" Oct 30 06:20:49.568959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:49.568943 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc84fe88-74f2-4174-ab6d-3b17d53805c2" path="/var/lib/kubelet/pods/cc84fe88-74f2-4174-ab6d-3b17d53805c2/volumes" Oct 30 06:20:56.584699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:56.584649 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr"] Oct 30 06:20:56.585311 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:56.585035 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" containerID="cri-o://3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113" gracePeriod=30 Oct 30 06:20:56.748552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:56.741723 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:20:57.113890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.113844 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst"] Oct 30 06:20:57.114736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.114708 5565 cpu_manager.go:414] "containerMap: removing container" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" Oct 30 06:20:57.114736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.114731 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" Oct 30 06:20:57.114916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.114904 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="99a0f929-f527-4920-ac51-6c0fbbfea9a9" containerName="istio-proxy" Oct 30 06:20:57.120168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.120143 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.130766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.130734 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.130917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.130855 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.130984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.130918 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xwk9\" (UniqueName: \"kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.136905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.136851 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:20:57.137258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.137230 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:20:57.139413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.139384 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst"] Oct 30 06:20:57.232606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.232567 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.232801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.232664 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.232801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.232684 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xwk9\" (UniqueName: \"kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.233389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.233367 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.235375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.235348 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.258266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.258233 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xwk9\" (UniqueName: \"kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.437745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.437651 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:20:57.754013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.753984 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst"] Oct 30 06:20:57.755924 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:20:57.755888 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975850ba_54ce_4b5e_80e0_3579ad0a571b.slice/crio-33b9068cd53d677604fb44f8fe14312303d9d27ceeefa22000cf3e5ab4034a9b WatchSource:0}: Error finding container 33b9068cd53d677604fb44f8fe14312303d9d27ceeefa22000cf3e5ab4034a9b: Status 404 returned error can't find the container with id 33b9068cd53d677604fb44f8fe14312303d9d27ceeefa22000cf3e5ab4034a9b Oct 30 06:20:57.836830 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:57.836795 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" event={"ID":"975850ba-54ce-4b5e-80e0-3579ad0a571b","Type":"ContainerStarted","Data":"33b9068cd53d677604fb44f8fe14312303d9d27ceeefa22000cf3e5ab4034a9b"} Oct 30 06:20:58.438308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.438225 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf"] Oct 30 06:20:58.442782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.442764 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.460547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.460495 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.460818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.460798 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.460915 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.460847 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42rvh\" (UniqueName: \"kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.464390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.464367 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf"] Oct 30 06:20:58.561934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.561898 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.561934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.561938 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42rvh\" (UniqueName: \"kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.562164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.562119 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.562721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.562701 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.564221 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.564197 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.586998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.586976 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42rvh\" (UniqueName: \"kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh\") pod \"maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.758716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.758688 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:20:58.853401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.853370 5565 generic.go:356] "Generic (PLEG): container finished" podID="975850ba-54ce-4b5e-80e0-3579ad0a571b" containerID="4bc9ae3e846cfb34979c8dedcd9552363285fe49cec5cb3b3b3d465cc37c354a" exitCode=0 Oct 30 06:20:58.853586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:58.853454 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" event={"ID":"975850ba-54ce-4b5e-80e0-3579ad0a571b","Type":"ContainerDied","Data":"4bc9ae3e846cfb34979c8dedcd9552363285fe49cec5cb3b3b3d465cc37c354a"} Oct 30 06:20:59.072841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:59.072809 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf"] Oct 30 06:20:59.073220 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:20:59.073194 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e374a1e_17bf_4aeb_9e3f_63a020be5c96.slice/crio-63c83baf33b436543d1e8b7dd4a7c31f9f46744e6c44e4395a7a8e99e27d5213 WatchSource:0}: Error finding container 63c83baf33b436543d1e8b7dd4a7c31f9f46744e6c44e4395a7a8e99e27d5213: Status 404 returned error can't find the container with id 63c83baf33b436543d1e8b7dd4a7c31f9f46744e6c44e4395a7a8e99e27d5213 Oct 30 06:20:59.863291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:59.863247 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" event={"ID":"2e374a1e-17bf-4aeb-9e3f-63a020be5c96","Type":"ContainerStarted","Data":"f400a7e545fe59b97827cca17b799a5a1ce8ba851d220c42434965cc83e3e939"} Oct 30 06:20:59.863755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:59.863317 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" event={"ID":"2e374a1e-17bf-4aeb-9e3f-63a020be5c96","Type":"ContainerStarted","Data":"63c83baf33b436543d1e8b7dd4a7c31f9f46744e6c44e4395a7a8e99e27d5213"} Oct 30 06:20:59.914583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:20:59.911891 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" podStartSLOduration=1.911871059 podStartE2EDuration="1.911871059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:20:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:20:59.906147419 +0000 UTC m=+2966.481834473" watchObservedRunningTime="2026-10-30 06:20:59.911871059 +0000 UTC m=+2966.487558113" Oct 30 06:21:00.038906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.038879 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:21:00.082668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.082467 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9\" (UniqueName: \"kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9\") pod \"975850ba-54ce-4b5e-80e0-3579ad0a571b\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " Oct 30 06:21:00.082668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.082610 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca\") pod \"975850ba-54ce-4b5e-80e0-3579ad0a571b\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " Oct 30 06:21:00.082991 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.082958 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca\") pod \"975850ba-54ce-4b5e-80e0-3579ad0a571b\" (UID: \"975850ba-54ce-4b5e-80e0-3579ad0a571b\") " Oct 30 06:21:00.083340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.083315 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca" pod "975850ba-54ce-4b5e-80e0-3579ad0a571b" (UID: "975850ba-54ce-4b5e-80e0-3579ad0a571b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:00.084007 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.083730 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.085981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.085284 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9" pod "975850ba-54ce-4b5e-80e0-3579ad0a571b" (UID: "975850ba-54ce-4b5e-80e0-3579ad0a571b"). InnerVolumeSpecName "kube-api-access-2xwk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:00.085981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.085475 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca" pod "975850ba-54ce-4b5e-80e0-3579ad0a571b" (UID: "975850ba-54ce-4b5e-80e0-3579ad0a571b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:00.184994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.184914 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xwk9\" (UniqueName: \"kubernetes.io/projected/975850ba-54ce-4b5e-80e0-3579ad0a571b-kube-api-access-2xwk9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.184994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.184941 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/975850ba-54ce-4b5e-80e0-3579ad0a571b-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:00.872757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.872722 5565 generic.go:356] "Generic (PLEG): container finished" podID="2e374a1e-17bf-4aeb-9e3f-63a020be5c96" containerID="f400a7e545fe59b97827cca17b799a5a1ce8ba851d220c42434965cc83e3e939" exitCode=0 Oct 30 06:21:00.873242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.872809 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" event={"ID":"2e374a1e-17bf-4aeb-9e3f-63a020be5c96","Type":"ContainerDied","Data":"f400a7e545fe59b97827cca17b799a5a1ce8ba851d220c42434965cc83e3e939"} Oct 30 06:21:00.874310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.874284 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" Oct 30 06:21:00.874310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.874308 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst" event={"ID":"975850ba-54ce-4b5e-80e0-3579ad0a571b","Type":"ContainerDied","Data":"33b9068cd53d677604fb44f8fe14312303d9d27ceeefa22000cf3e5ab4034a9b"} Oct 30 06:21:00.874618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:00.874354 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33b9068cd53d677604fb44f8fe14312303d9d27ceeefa22000cf3e5ab4034a9b" Oct 30 06:21:01.877404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.877322 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:21:01.882448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.882418 5565 generic.go:356] "Generic (PLEG): container finished" podID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerID="3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113" exitCode=0 Oct 30 06:21:01.882663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.882470 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" Oct 30 06:21:01.882663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.882503 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" event={"ID":"f0f65c9b-058b-4e1b-9dff-54d86121798c","Type":"ContainerDied","Data":"3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113"} Oct 30 06:21:01.882663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.882572 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr" event={"ID":"f0f65c9b-058b-4e1b-9dff-54d86121798c","Type":"ContainerDied","Data":"cc6fad37f63ce223ab04781a1488aa50d7d579f20eb833d32143cf05cedf616a"} Oct 30 06:21:01.882663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.882599 5565 scope.go:122] "RemoveContainer" containerID="3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113" Oct 30 06:21:01.893748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.893722 5565 scope.go:122] "RemoveContainer" containerID="3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113" Oct 30 06:21:01.894111 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:01.894088 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113\": container with ID starting with 3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113 not found: ID does not exist" containerID="3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113" Oct 30 06:21:01.894185 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.894121 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113"} err="failed to get container status \"3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113\": rpc error: code = NotFound desc = could not find container \"3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113\": container with ID starting with 3e9b3ed94c1a06cfcaeb72f2c8efd43023e7bd51e0b293770e4ad1070a866113 not found: ID does not exist" Oct 30 06:21:01.910910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.910860 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.910910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.910910 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911019 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911078 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911109 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911177 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911247 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911316 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911421 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data\") pod \"f0f65c9b-058b-4e1b-9dff-54d86121798c\" (UID: \"f0f65c9b-058b-4e1b-9dff-54d86121798c\") " Oct 30 06:21:01.911761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911689 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.911761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911704 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.912003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.911892 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.912264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.912216 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:01.912371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.912268 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.912371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.912287 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.912371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.912300 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:01.912572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.912548 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:01.913420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.913400 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:01.913908 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.913882 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:01.914003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.913922 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "kube-api-access-sxx2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:01.914064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:01.914024 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy" pod "f0f65c9b-058b-4e1b-9dff-54d86121798c" (UID: "f0f65c9b-058b-4e1b-9dff-54d86121798c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:02.013761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.013721 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.013761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.013760 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.014003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.013773 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sxx2h\" (UniqueName: \"kubernetes.io/projected/f0f65c9b-058b-4e1b-9dff-54d86121798c-kube-api-access-sxx2h\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.014003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.013787 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0f65c9b-058b-4e1b-9dff-54d86121798c-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.014003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.013798 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.014003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.013808 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0f65c9b-058b-4e1b-9dff-54d86121798c-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.044775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.044736 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:21:02.115187 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.115148 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca\") pod \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " Oct 30 06:21:02.115393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.115335 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca\") pod \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " Oct 30 06:21:02.115467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.115440 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh\" (UniqueName: \"kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh\") pod \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\" (UID: \"2e374a1e-17bf-4aeb-9e3f-63a020be5c96\") " Oct 30 06:21:02.115637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.115611 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca" pod "2e374a1e-17bf-4aeb-9e3f-63a020be5c96" (UID: "2e374a1e-17bf-4aeb-9e3f-63a020be5c96"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:02.116165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.116139 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.117799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.117767 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca" pod "2e374a1e-17bf-4aeb-9e3f-63a020be5c96" (UID: "2e374a1e-17bf-4aeb-9e3f-63a020be5c96"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:02.117929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.117767 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh" pod "2e374a1e-17bf-4aeb-9e3f-63a020be5c96" (UID: "2e374a1e-17bf-4aeb-9e3f-63a020be5c96"). InnerVolumeSpecName "kube-api-access-42rvh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:02.217230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.217194 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.217230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.217226 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42rvh\" (UniqueName: \"kubernetes.io/projected/2e374a1e-17bf-4aeb-9e3f-63a020be5c96-kube-api-access-42rvh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:02.244167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.244130 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr"] Oct 30 06:21:02.248422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.248386 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-1b4b7a-openshift-default-bc5987685-rsdlr"] Oct 30 06:21:02.891561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.891434 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" Oct 30 06:21:02.891561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.891472 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf" event={"ID":"2e374a1e-17bf-4aeb-9e3f-63a020be5c96","Type":"ContainerDied","Data":"63c83baf33b436543d1e8b7dd4a7c31f9f46744e6c44e4395a7a8e99e27d5213"} Oct 30 06:21:02.892090 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:02.891568 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="63c83baf33b436543d1e8b7dd4a7c31f9f46744e6c44e4395a7a8e99e27d5213" Oct 30 06:21:03.569295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:03.569255 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" path="/var/lib/kubelet/pods/f0f65c9b-058b-4e1b-9dff-54d86121798c/volumes" Oct 30 06:21:04.503216 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:04.503187 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:21:04.691106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:04.691072 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:21:05.539963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.539922 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g"] Oct 30 06:21:05.540693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540675 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" Oct 30 06:21:05.540693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540693 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" Oct 30 06:21:05.540775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540712 5565 cpu_manager.go:414] "containerMap: removing container" podUID="975850ba-54ce-4b5e-80e0-3579ad0a571b" containerName="revoke-keys" Oct 30 06:21:05.540775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540718 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="975850ba-54ce-4b5e-80e0-3579ad0a571b" containerName="revoke-keys" Oct 30 06:21:05.540775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540754 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2e374a1e-17bf-4aeb-9e3f-63a020be5c96" containerName="revoke-keys" Oct 30 06:21:05.540775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540759 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e374a1e-17bf-4aeb-9e3f-63a020be5c96" containerName="revoke-keys" Oct 30 06:21:05.540911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540877 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e374a1e-17bf-4aeb-9e3f-63a020be5c96" containerName="revoke-keys" Oct 30 06:21:05.540911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540892 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0f65c9b-058b-4e1b-9dff-54d86121798c" containerName="istio-proxy" Oct 30 06:21:05.540911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.540899 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="975850ba-54ce-4b5e-80e0-3579ad0a571b" containerName="revoke-keys" Oct 30 06:21:05.544761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.544735 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.571693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.571656 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g"] Oct 30 06:21:05.574732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.574691 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:05.574732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.574737 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:21:05.679748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.679701 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-888x7\" (UniqueName: \"kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.679959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.679858 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.680127 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.680105 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.781984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.781942 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.782183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.781995 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-888x7\" (UniqueName: \"kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.782183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.782072 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.782767 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.782742 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.784436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.784411 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.808257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.808167 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-888x7\" (UniqueName: \"kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7\") pod \"maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:05.859700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:05.859656 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:06.172478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:06.172439 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g"] Oct 30 06:21:06.174179 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:21:06.174121 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7566c8e_9567_4891_8e5c_cdc1221ea5df.slice/crio-033aa4aad5d0387780ac85ab974181fcb0fd87f73af67dbed7fa91e7c5c190d0 WatchSource:0}: Error finding container 033aa4aad5d0387780ac85ab974181fcb0fd87f73af67dbed7fa91e7c5c190d0: Status 404 returned error can't find the container with id 033aa4aad5d0387780ac85ab974181fcb0fd87f73af67dbed7fa91e7c5c190d0 Oct 30 06:21:06.934715 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:06.934677 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" event={"ID":"b7566c8e-9567-4891-8e5c-cdc1221ea5df","Type":"ContainerStarted","Data":"cb9262272a3bf2028dc5b96a4c2f15be0545442098caaae529ef6a7c91526dae"} Oct 30 06:21:06.935182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:06.934747 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" event={"ID":"b7566c8e-9567-4891-8e5c-cdc1221ea5df","Type":"ContainerStarted","Data":"033aa4aad5d0387780ac85ab974181fcb0fd87f73af67dbed7fa91e7c5c190d0"} Oct 30 06:21:06.980154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:06.980092 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" podStartSLOduration=1.980078591 podStartE2EDuration="1.980078591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:06.97858249 +0000 UTC m=+2973.554269546" watchObservedRunningTime="2026-10-30 06:21:06.980078591 +0000 UTC m=+2973.555765644" Oct 30 06:21:07.467951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:07.467913 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d"] Oct 30 06:21:07.495942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:07.495905 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh"] Oct 30 06:21:07.944433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:07.944393 5565 generic.go:356] "Generic (PLEG): container finished" podID="b7566c8e-9567-4891-8e5c-cdc1221ea5df" containerID="cb9262272a3bf2028dc5b96a4c2f15be0545442098caaae529ef6a7c91526dae" exitCode=0 Oct 30 06:21:07.944874 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:07.944477 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" event={"ID":"b7566c8e-9567-4891-8e5c-cdc1221ea5df","Type":"ContainerDied","Data":"cb9262272a3bf2028dc5b96a4c2f15be0545442098caaae529ef6a7c91526dae"} Oct 30 06:21:09.035014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.034975 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56"] Oct 30 06:21:09.046776 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.046742 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cf777683-5jw56"] Oct 30 06:21:09.110154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.110130 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:09.258612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.258579 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca\") pod \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " Oct 30 06:21:09.258772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.258710 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7\" (UniqueName: \"kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7\") pod \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " Oct 30 06:21:09.258772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.258763 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca\") pod \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\" (UID: \"b7566c8e-9567-4891-8e5c-cdc1221ea5df\") " Oct 30 06:21:09.258949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.258925 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca" pod "b7566c8e-9567-4891-8e5c-cdc1221ea5df" (UID: "b7566c8e-9567-4891-8e5c-cdc1221ea5df"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:09.259350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.259329 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.260831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.260793 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca" pod "b7566c8e-9567-4891-8e5c-cdc1221ea5df" (UID: "b7566c8e-9567-4891-8e5c-cdc1221ea5df"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:09.260951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.260865 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7" pod "b7566c8e-9567-4891-8e5c-cdc1221ea5df" (UID: "b7566c8e-9567-4891-8e5c-cdc1221ea5df"). InnerVolumeSpecName "kube-api-access-888x7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:09.360712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.360682 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-888x7\" (UniqueName: \"kubernetes.io/projected/b7566c8e-9567-4891-8e5c-cdc1221ea5df-kube-api-access-888x7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.360712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.360710 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b7566c8e-9567-4891-8e5c-cdc1221ea5df-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:09.569106 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.569030 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="219e908c-5a9a-4b73-84db-359cf49e5bae" path="/var/lib/kubelet/pods/219e908c-5a9a-4b73-84db-359cf49e5bae/volumes" Oct 30 06:21:09.961568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.961471 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" Oct 30 06:21:09.961568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.961517 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g" event={"ID":"b7566c8e-9567-4891-8e5c-cdc1221ea5df","Type":"ContainerDied","Data":"033aa4aad5d0387780ac85ab974181fcb0fd87f73af67dbed7fa91e7c5c190d0"} Oct 30 06:21:09.961764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:09.961589 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="033aa4aad5d0387780ac85ab974181fcb0fd87f73af67dbed7fa91e7c5c190d0" Oct 30 06:21:11.258079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:11.258044 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:21:12.469088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:12.469040 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerName="payload-processing" containerID="cri-o://98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a" gracePeriod=25 Oct 30 06:21:12.496969 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:12.496907 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerName="payload-pre-processing" containerID="cri-o://23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8" gracePeriod=25 Oct 30 06:21:15.030630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:15.030589 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b"] Oct 30 06:21:15.044410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:15.044369 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c59c6eb0-cbb9b"] Oct 30 06:21:15.568660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:15.568621 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bfa5f9-77e2-478e-bfa1-4d0163b757ab" path="/var/lib/kubelet/pods/07bfa5f9-77e2-478e-bfa1-4d0163b757ab/volumes" Oct 30 06:21:17.533042 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:17.532989 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:21:17.929706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:17.929604 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:17.929706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:17.929627 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:26.031710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:26.031673 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx"] Oct 30 06:21:26.045019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:26.044986 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d5c62456-qdfsx"] Oct 30 06:21:27.440462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:27.440426 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf"] Oct 30 06:21:27.440916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:27.440694 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" containerID="cri-o://71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78" gracePeriod=30 Oct 30 06:21:27.569934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:27.569903 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c3cb5b4-4c8a-4098-a3d9-538bfcf69964" path="/var/lib/kubelet/pods/3c3cb5b4-4c8a-4098-a3d9-538bfcf69964/volumes" Oct 30 06:21:27.929589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:27.929552 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:27.929837 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:27.929813 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:21:30.783810 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:30.783776 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl"] Oct 30 06:21:31.923906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.923868 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4"] Oct 30 06:21:31.924838 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.924782 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b7566c8e-9567-4891-8e5c-cdc1221ea5df" containerName="revoke-keys" Oct 30 06:21:31.924838 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.924805 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7566c8e-9567-4891-8e5c-cdc1221ea5df" containerName="revoke-keys" Oct 30 06:21:31.925087 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.925035 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7566c8e-9567-4891-8e5c-cdc1221ea5df" containerName="revoke-keys" Oct 30 06:21:31.930520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.930500 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:31.949310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.949282 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4"] Oct 30 06:21:31.960072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.960008 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d4bace\"" Oct 30 06:21:31.960245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:31.960152 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d4bace-dockercfg-9vj6q\"" Oct 30 06:21:32.000085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.000052 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdn9r\" (UniqueName: \"kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r\") pod \"payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.000298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.000236 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.065603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.065550 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd"] Oct 30 06:21:32.070147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.070123 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.090372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.090342 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd"] Oct 30 06:21:32.101977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.101942 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.102161 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.102107 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdn9r\" (UniqueName: \"kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r\") pod \"payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.102918 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.102889 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.127626 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.127592 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdn9r\" (UniqueName: \"kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r\") pod \"payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.203147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.203061 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.203147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.203136 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqwjn\" (UniqueName: \"kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.245439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.245404 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:32.305210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.304714 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.305210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.304830 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqwjn\" (UniqueName: \"kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.306312 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.306283 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.346551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.342750 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqwjn\" (UniqueName: \"kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn\") pod \"payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.384877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.384838 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:32.565842 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.565804 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4"] Oct 30 06:21:32.710519 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.710137 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd"] Oct 30 06:21:32.828308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.828015 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:21:32.887722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.887686 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:21:32.916981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.916778 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume\") pod \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " Oct 30 06:21:32.916981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.916948 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5\" (UniqueName: \"kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5\") pod \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\" (UID: \"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c\") " Oct 30 06:21:32.918558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.918496 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume" pod "8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" (UID: "8bf15dfe-a041-4b05-9e4b-7b988a24ec4c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:32.921122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.921083 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5" pod "8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" (UID: "8bf15dfe-a041-4b05-9e4b-7b988a24ec4c"). InnerVolumeSpecName "kube-api-access-vs7m5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:32.930363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:32.930330 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:21:33.018615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.018576 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.018802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.018652 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.018802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.018697 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.018928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.018826 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.018928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.018868 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.018928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.018918 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume\") pod \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " Oct 30 06:21:33.019107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019026 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019191 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019227 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246\" (UniqueName: \"kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246\") pod \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\" (UID: \"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a\") " Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019254 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019272 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019283 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo\") pod \"d3c91cec-c5fd-42a0-9135-3210a8261b02\" (UID: \"d3c91cec-c5fd-42a0-9135-3210a8261b02\") " Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.019926 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020157 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020178 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020192 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs7m5\" (UniqueName: \"kubernetes.io/projected/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c-kube-api-access-vs7m5\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020206 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020257 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume" pod "c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" (UID: "c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020395 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020505 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:33.020639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.020577 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:33.022658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.022631 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246" pod "c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" (UID: "c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a"). InnerVolumeSpecName "kube-api-access-kj246". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:33.022930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.022869 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:21:33.023967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.023936 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:33.024427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.024400 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:21:33.024692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.024672 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj" pod "d3c91cec-c5fd-42a0-9135-3210a8261b02" (UID: "d3c91cec-c5fd-42a0-9135-3210a8261b02"). InnerVolumeSpecName "kube-api-access-67zdj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:33.121338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121295 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67zdj\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-kube-api-access-67zdj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121335 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d3c91cec-c5fd-42a0-9135-3210a8261b02-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121345 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121354 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121363 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121375 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121383 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj246\" (UniqueName: \"kubernetes.io/projected/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a-kube-api-access-kj246\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121391 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.121628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.121400 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d3c91cec-c5fd-42a0-9135-3210a8261b02-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:33.134190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.134154 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" event={"ID":"4a03a379-915f-49d6-98e5-a6aaa7275e7b","Type":"ContainerStarted","Data":"59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9"} Oct 30 06:21:33.134368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.134219 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" event={"ID":"4a03a379-915f-49d6-98e5-a6aaa7275e7b","Type":"ContainerStarted","Data":"958bd4ffe8d3ffe1195d64c1d066101cb3430ad809b399bd17ef6f60342eb2d4"} Oct 30 06:21:33.134368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.134248 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:33.135999 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.135973 5565 generic.go:356] "Generic (PLEG): container finished" podID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerID="98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a" exitCode=0 Oct 30 06:21:33.136165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.136011 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" Oct 30 06:21:33.136165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.136040 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" event={"ID":"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c","Type":"ContainerDied","Data":"98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a"} Oct 30 06:21:33.136165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.136097 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d" event={"ID":"8bf15dfe-a041-4b05-9e4b-7b988a24ec4c","Type":"ContainerDied","Data":"6112dfbf2e365d9b8e34fd276228785e6afabc92eec32586ecc43e1cc9420bae"} Oct 30 06:21:33.136165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.136128 5565 scope.go:122] "RemoveContainer" containerID="98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a" Oct 30 06:21:33.137701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.137666 5565 generic.go:356] "Generic (PLEG): container finished" podID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerID="23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8" exitCode=0 Oct 30 06:21:33.137840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.137725 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" Oct 30 06:21:33.137840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.137775 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" event={"ID":"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a","Type":"ContainerDied","Data":"23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8"} Oct 30 06:21:33.137840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.137816 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh" event={"ID":"c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a","Type":"ContainerDied","Data":"e7730201baa1473855dd8d4d99e23d696f61d71c45d53278e507acd0fa4629e5"} Oct 30 06:21:33.142488 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.142455 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" event={"ID":"cbcd69ad-b321-40f0-a2b8-691dcaab5011","Type":"ContainerStarted","Data":"fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6"} Oct 30 06:21:33.142702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.142515 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" event={"ID":"cbcd69ad-b321-40f0-a2b8-691dcaab5011","Type":"ContainerStarted","Data":"c0fa4816008718cd48b972c1da578d20a5ccf65e836839197ee3e166398bbc42"} Oct 30 06:21:33.142702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.142569 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:33.145484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.145409 5565 generic.go:356] "Generic (PLEG): container finished" podID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerID="71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78" exitCode=0 Oct 30 06:21:33.145484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.145453 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" event={"ID":"d3c91cec-c5fd-42a0-9135-3210a8261b02","Type":"ContainerDied","Data":"71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78"} Oct 30 06:21:33.145484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.145482 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" event={"ID":"d3c91cec-c5fd-42a0-9135-3210a8261b02","Type":"ContainerDied","Data":"aac88cf9fafd019586c540588d35be72c6954b0baa4156c588a14449e3fb1985"} Oct 30 06:21:33.145647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.145575 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf" Oct 30 06:21:33.176296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.176268 5565 scope.go:122] "RemoveContainer" containerID="98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a" Oct 30 06:21:33.176698 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:33.176675 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a\": container with ID starting with 98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a not found: ID does not exist" containerID="98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a" Oct 30 06:21:33.176774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.176713 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a"} err="failed to get container status \"98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a\": rpc error: code = NotFound desc = could not find container \"98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a\": container with ID starting with 98fb844c19056335e60e5700ddc327cb6e32bde852bdbff498718cd0e345635a not found: ID does not exist" Oct 30 06:21:33.176774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.176739 5565 scope.go:122] "RemoveContainer" containerID="23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8" Oct 30 06:21:33.181746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.181697 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" podStartSLOduration=1.181683846 podStartE2EDuration="1.181683846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:33.179225733 +0000 UTC m=+2999.754912787" watchObservedRunningTime="2026-10-30 06:21:33.181683846 +0000 UTC m=+2999.757370899" Oct 30 06:21:33.192786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.192761 5565 scope.go:122] "RemoveContainer" containerID="23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8" Oct 30 06:21:33.193169 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:33.193141 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8\": container with ID starting with 23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8 not found: ID does not exist" containerID="23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8" Oct 30 06:21:33.193249 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.193183 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8"} err="failed to get container status \"23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8\": rpc error: code = NotFound desc = could not find container \"23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8\": container with ID starting with 23e48506d3b20bd4db37da0bf87946bded2b46ab16bec820514856efb6db07a8 not found: ID does not exist" Oct 30 06:21:33.193249 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.193212 5565 scope.go:122] "RemoveContainer" containerID="71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78" Oct 30 06:21:33.206082 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.206058 5565 scope.go:122] "RemoveContainer" containerID="71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78" Oct 30 06:21:33.206463 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:33.206436 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78\": container with ID starting with 71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78 not found: ID does not exist" containerID="71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78" Oct 30 06:21:33.206633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.206476 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78"} err="failed to get container status \"71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78\": rpc error: code = NotFound desc = could not find container \"71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78\": container with ID starting with 71374d98d089ac08700afc43837fdc040df8bb4754e9609b88501f658fd88a78 not found: ID does not exist" Oct 30 06:21:33.222189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.222155 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh"] Oct 30 06:21:33.228688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.228661 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-02ceed-7bb8bd9d4c-snjzh"] Oct 30 06:21:33.272875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.272771 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" podStartSLOduration=2.2727565849999998 podStartE2EDuration="2.272756585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:21:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:21:33.271087181 +0000 UTC m=+2999.846774238" watchObservedRunningTime="2026-10-30 06:21:33.272756585 +0000 UTC m=+2999.848443638" Oct 30 06:21:33.312861 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.312791 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d"] Oct 30 06:21:33.317020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.316988 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-02ceed-5bc465ffbd-wwj6d"] Oct 30 06:21:33.358592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.358523 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf"] Oct 30 06:21:33.363500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.363451 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-ns5sf"] Oct 30 06:21:33.570874 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.570791 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" path="/var/lib/kubelet/pods/8bf15dfe-a041-4b05-9e4b-7b988a24ec4c/volumes" Oct 30 06:21:33.571355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.571339 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" path="/var/lib/kubelet/pods/c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a/volumes" Oct 30 06:21:33.571964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:33.571949 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" path="/var/lib/kubelet/pods/d3c91cec-c5fd-42a0-9135-3210a8261b02/volumes" Oct 30 06:21:35.033281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:35.033234 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh"] Oct 30 06:21:35.047452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:35.047416 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4b66648f-88nsh"] Oct 30 06:21:35.568660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:35.568622 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce571bd4-e31b-41e4-8e3b-3a3459e765f1" path="/var/lib/kubelet/pods/ce571bd4-e31b-41e4-8e3b-3a3459e765f1/volumes" Oct 30 06:21:35.784801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:35.784756 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" podUID="bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" containerName="payload-pre-processing" containerID="cri-o://8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7" gracePeriod=25 Oct 30 06:21:36.102774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.102747 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:21:36.173200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.173115 5565 generic.go:356] "Generic (PLEG): container finished" podID="bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" containerID="8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7" exitCode=0 Oct 30 06:21:36.173200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.173173 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" Oct 30 06:21:36.173479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.173198 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" event={"ID":"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d","Type":"ContainerDied","Data":"8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7"} Oct 30 06:21:36.173479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.173265 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl" event={"ID":"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d","Type":"ContainerDied","Data":"780df681229c5bc30cf0194bc0deb99d4a71517359246d25dc778fb754fcbead"} Oct 30 06:21:36.173479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.173297 5565 scope.go:122] "RemoveContainer" containerID="8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7" Oct 30 06:21:36.186440 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.186414 5565 scope.go:122] "RemoveContainer" containerID="8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7" Oct 30 06:21:36.186831 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:36.186807 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7\": container with ID starting with 8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7 not found: ID does not exist" containerID="8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7" Oct 30 06:21:36.186943 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.186839 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7"} err="failed to get container status \"8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7\": rpc error: code = NotFound desc = could not find container \"8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7\": container with ID starting with 8f9835416abde70c3061462e4dc07944aab124f00583e6fb6d812048f952e2d7 not found: ID does not exist" Oct 30 06:21:36.275547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.275505 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn\" (UniqueName: \"kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn\") pod \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " Oct 30 06:21:36.275726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.275595 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume\") pod \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\" (UID: \"bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d\") " Oct 30 06:21:36.276005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.275982 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume" pod "bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" (UID: "bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:36.276151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.276133 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.277811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.277789 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn" pod "bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" (UID: "bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d"). InnerVolumeSpecName "kube-api-access-rhvbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:36.377256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.377218 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhvbn\" (UniqueName: \"kubernetes.io/projected/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d-kube-api-access-rhvbn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:36.388939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.388883 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst"] Oct 30 06:21:36.393758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.393716 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-02ceed-62fst"] Oct 30 06:21:36.571898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.571862 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl"] Oct 30 06:21:36.575057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:36.575020 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-718730-6d4884f4c5-9kmnl"] Oct 30 06:21:37.568943 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:37.568909 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="975850ba-54ce-4b5e-80e0-3579ad0a571b" path="/var/lib/kubelet/pods/975850ba-54ce-4b5e-80e0-3579ad0a571b/volumes" Oct 30 06:21:37.569377 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:37.569362 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" path="/var/lib/kubelet/pods/bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d/volumes" Oct 30 06:21:41.162698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.162652 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7"] Oct 30 06:21:41.163261 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.162889 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" podUID="baaa1333-908d-4961-8812-675996545881" containerName="payload-processing" containerID="cri-o://0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68" gracePeriod=30 Oct 30 06:21:41.178870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.178837 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk"] Oct 30 06:21:41.179195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.179160 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" podUID="589bbae7-449b-4aa2-9fa2-20a5b48d09b1" containerName="payload-pre-processing" containerID="cri-o://37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399" gracePeriod=30 Oct 30 06:21:41.234572 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:41.234491 5565 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaaa1333_908d_4961_8812_675996545881.slice/crio-0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68.scope\": RecentStats: unable to find data in memory cache]" Oct 30 06:21:41.550506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.549768 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:21:41.585597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.564947 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume\") pod \"baaa1333-908d-4961-8812-675996545881\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " Oct 30 06:21:41.585597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.565143 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6\" (UniqueName: \"kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6\") pod \"baaa1333-908d-4961-8812-675996545881\" (UID: \"baaa1333-908d-4961-8812-675996545881\") " Oct 30 06:21:41.591549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.589829 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:21:41.598585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.598478 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6" pod "baaa1333-908d-4961-8812-675996545881" (UID: "baaa1333-908d-4961-8812-675996545881"). InnerVolumeSpecName "kube-api-access-9kzv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:41.598982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.598954 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume" pod "baaa1333-908d-4961-8812-675996545881" (UID: "baaa1333-908d-4961-8812-675996545881"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:41.667129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.667090 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume\") pod \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " Oct 30 06:21:41.667334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.667203 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv\" (UniqueName: \"kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv\") pod \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\" (UID: \"589bbae7-449b-4aa2-9fa2-20a5b48d09b1\") " Oct 30 06:21:41.667858 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.667599 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume" pod "589bbae7-449b-4aa2-9fa2-20a5b48d09b1" (UID: "589bbae7-449b-4aa2-9fa2-20a5b48d09b1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:41.668659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.668384 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baaa1333-908d-4961-8812-675996545881-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:41.668659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.668413 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:41.668659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.668428 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kzv6\" (UniqueName: \"kubernetes.io/projected/baaa1333-908d-4961-8812-675996545881-kube-api-access-9kzv6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:41.669810 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.669787 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv" pod "589bbae7-449b-4aa2-9fa2-20a5b48d09b1" (UID: "589bbae7-449b-4aa2-9fa2-20a5b48d09b1"). InnerVolumeSpecName "kube-api-access-8zhjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:41.769870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.769835 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zhjv\" (UniqueName: \"kubernetes.io/projected/589bbae7-449b-4aa2-9fa2-20a5b48d09b1-kube-api-access-8zhjv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:41.790622 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.790585 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf"] Oct 30 06:21:41.793459 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:41.793435 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5a591d-s5pcf"] Oct 30 06:21:42.234456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.234418 5565 generic.go:356] "Generic (PLEG): container finished" podID="589bbae7-449b-4aa2-9fa2-20a5b48d09b1" containerID="37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399" exitCode=0 Oct 30 06:21:42.234456 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.234470 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" Oct 30 06:21:42.235066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.234504 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" event={"ID":"589bbae7-449b-4aa2-9fa2-20a5b48d09b1","Type":"ContainerDied","Data":"37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399"} Oct 30 06:21:42.235066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.234598 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk" event={"ID":"589bbae7-449b-4aa2-9fa2-20a5b48d09b1","Type":"ContainerDied","Data":"909e7732b4026c23f6fc7e971182d71efb872744bed253773e7e769a0bc2c8ce"} Oct 30 06:21:42.235066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.234637 5565 scope.go:122] "RemoveContainer" containerID="37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399" Oct 30 06:21:42.236223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.236201 5565 generic.go:356] "Generic (PLEG): container finished" podID="baaa1333-908d-4961-8812-675996545881" containerID="0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68" exitCode=0 Oct 30 06:21:42.236339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.236257 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" Oct 30 06:21:42.236339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.236256 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" event={"ID":"baaa1333-908d-4961-8812-675996545881","Type":"ContainerDied","Data":"0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68"} Oct 30 06:21:42.236339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.236309 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7" event={"ID":"baaa1333-908d-4961-8812-675996545881","Type":"ContainerDied","Data":"153ec704aad21a42e48db7b85ac560601727b7597346549b337604738212fb68"} Oct 30 06:21:42.247982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.247964 5565 scope.go:122] "RemoveContainer" containerID="37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399" Oct 30 06:21:42.248278 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:42.248251 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399\": container with ID starting with 37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399 not found: ID does not exist" containerID="37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399" Oct 30 06:21:42.248351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.248289 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399"} err="failed to get container status \"37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399\": rpc error: code = NotFound desc = could not find container \"37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399\": container with ID starting with 37c42b3d647cddaf1b57b5e88696094415687562d89dc04545f64e3261869399 not found: ID does not exist" Oct 30 06:21:42.248351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.248314 5565 scope.go:122] "RemoveContainer" containerID="0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68" Oct 30 06:21:42.260439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.260420 5565 scope.go:122] "RemoveContainer" containerID="0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68" Oct 30 06:21:42.260758 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:42.260724 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68\": container with ID starting with 0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68 not found: ID does not exist" containerID="0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68" Oct 30 06:21:42.260877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.260766 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68"} err="failed to get container status \"0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68\": rpc error: code = NotFound desc = could not find container \"0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68\": container with ID starting with 0d1aefd33c7d2715b9e1bf6a0ca4126221d48a21e74b903d7fe52d4a70fb8b68 not found: ID does not exist" Oct 30 06:21:42.297483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.297442 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7"] Oct 30 06:21:42.311624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.311593 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5a591d-6785c8b66-6xln7"] Oct 30 06:21:42.338111 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.338081 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk"] Oct 30 06:21:42.344864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:42.344830 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5a591d-7d6db867bc-qh9hk"] Oct 30 06:21:43.569049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:43.569010 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e374a1e-17bf-4aeb-9e3f-63a020be5c96" path="/var/lib/kubelet/pods/2e374a1e-17bf-4aeb-9e3f-63a020be5c96/volumes" Oct 30 06:21:43.569498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:43.569482 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="589bbae7-449b-4aa2-9fa2-20a5b48d09b1" path="/var/lib/kubelet/pods/589bbae7-449b-4aa2-9fa2-20a5b48d09b1/volumes" Oct 30 06:21:43.569945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:43.569930 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="baaa1333-908d-4961-8812-675996545881" path="/var/lib/kubelet/pods/baaa1333-908d-4961-8812-675996545881/volumes" Oct 30 06:21:43.810737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:43.810697 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s"] Oct 30 06:21:43.810943 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:43.810910 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" podUID="4ef84b6f-f535-4625-a871-4bc7597e373b" containerName="payload-processing" containerID="cri-o://75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db" gracePeriod=30 Oct 30 06:21:44.124411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.124380 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:21:44.159595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.159562 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:21:44.159759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.159643 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:21:44.228759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.220234 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume\") pod \"4ef84b6f-f535-4625-a871-4bc7597e373b\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " Oct 30 06:21:44.256553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.226558 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume" pod "4ef84b6f-f535-4625-a871-4bc7597e373b" (UID: "4ef84b6f-f535-4625-a871-4bc7597e373b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:44.256553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.235810 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz\" (UniqueName: \"kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz\") pod \"4ef84b6f-f535-4625-a871-4bc7597e373b\" (UID: \"4ef84b6f-f535-4625-a871-4bc7597e373b\") " Oct 30 06:21:44.286612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.258101 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4ef84b6f-f535-4625-a871-4bc7597e373b-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.293849 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.293762 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz" pod "4ef84b6f-f535-4625-a871-4bc7597e373b" (UID: "4ef84b6f-f535-4625-a871-4bc7597e373b"). InnerVolumeSpecName "kube-api-access-xl6rz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:44.384949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.384851 5565 generic.go:356] "Generic (PLEG): container finished" podID="4ef84b6f-f535-4625-a871-4bc7597e373b" containerID="75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db" exitCode=0 Oct 30 06:21:44.385447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.385229 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" event={"ID":"4ef84b6f-f535-4625-a871-4bc7597e373b","Type":"ContainerDied","Data":"75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db"} Oct 30 06:21:44.385447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.385373 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" event={"ID":"4ef84b6f-f535-4625-a871-4bc7597e373b","Type":"ContainerDied","Data":"005c31c5f417fc9820059927590d824ee59a44655f3344a8d0b39c541f4ec50a"} Oct 30 06:21:44.385447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.385414 5565 scope.go:122] "RemoveContainer" containerID="75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db" Oct 30 06:21:44.386384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.386343 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s" Oct 30 06:21:44.389272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.389240 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl6rz\" (UniqueName: \"kubernetes.io/projected/4ef84b6f-f535-4625-a871-4bc7597e373b-kube-api-access-xl6rz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:44.406146 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.406116 5565 scope.go:122] "RemoveContainer" containerID="75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db" Oct 30 06:21:44.406494 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:44.406469 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db\": container with ID starting with 75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db not found: ID does not exist" containerID="75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db" Oct 30 06:21:44.406609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.406504 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db"} err="failed to get container status \"75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db\": rpc error: code = NotFound desc = could not find container \"75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db\": container with ID starting with 75f6c13cd812776e758e61a7bbe49ca5a5671d56873ad4e1cf421de96ae544db not found: ID does not exist" Oct 30 06:21:44.446825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.446785 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s"] Oct 30 06:21:44.456097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.456054 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-fb9345-65f8f4d8dd-fsv7s"] Oct 30 06:21:44.538170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.538134 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g"] Oct 30 06:21:44.545093 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.545050 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fb9345-bjx4g"] Oct 30 06:21:44.654338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.654198 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd"] Oct 30 06:21:44.655123 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655100 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" containerName="payload-pre-processing" Oct 30 06:21:44.655175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655125 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" containerName="payload-pre-processing" Oct 30 06:21:44.655175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655145 5565 cpu_manager.go:414] "containerMap: removing container" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerName="payload-processing" Oct 30 06:21:44.655175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655151 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerName="payload-processing" Oct 30 06:21:44.655175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655174 5565 cpu_manager.go:414] "containerMap: removing container" podUID="589bbae7-449b-4aa2-9fa2-20a5b48d09b1" containerName="payload-pre-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655183 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="589bbae7-449b-4aa2-9fa2-20a5b48d09b1" containerName="payload-pre-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655202 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4ef84b6f-f535-4625-a871-4bc7597e373b" containerName="payload-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655210 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ef84b6f-f535-4625-a871-4bc7597e373b" containerName="payload-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655270 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerName="payload-pre-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655279 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerName="payload-pre-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655297 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655304 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655330 5565 cpu_manager.go:414] "containerMap: removing container" podUID="baaa1333-908d-4961-8812-675996545881" containerName="payload-processing" Oct 30 06:21:44.655422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655337 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="baaa1333-908d-4961-8812-675996545881" containerName="payload-processing" Oct 30 06:21:44.656190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655517 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d89bf4-20bc-43ef-b5b9-6fb561a77b0a" containerName="payload-pre-processing" Oct 30 06:21:44.656190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.655630 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3c91cec-c5fd-42a0-9135-3210a8261b02" containerName="istio-proxy" Oct 30 06:21:44.656825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.656791 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="baaa1333-908d-4961-8812-675996545881" containerName="payload-processing" Oct 30 06:21:44.656825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.656826 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd6f0329-e9ac-4d7f-8e97-1a1b8b5b196d" containerName="payload-pre-processing" Oct 30 06:21:44.656998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.656834 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bf15dfe-a041-4b05-9e4b-7b988a24ec4c" containerName="payload-processing" Oct 30 06:21:44.656998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.656845 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="589bbae7-449b-4aa2-9fa2-20a5b48d09b1" containerName="payload-pre-processing" Oct 30 06:21:44.656998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.656858 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ef84b6f-f535-4625-a871-4bc7597e373b" containerName="payload-processing" Oct 30 06:21:44.662689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.662659 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.671577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.671536 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd"] Oct 30 06:21:44.683580 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.679667 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:44.683580 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.679830 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:21:44.694025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.693983 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt9kj\" (UniqueName: \"kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.694301 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.694278 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.694382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.694315 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.795329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.795271 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.795329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.795322 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.795718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.795455 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt9kj\" (UniqueName: \"kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.796138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.796104 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.798212 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.798176 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.821749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.821707 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt9kj\" (UniqueName: \"kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj\") pod \"maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:44.979832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:44.979731 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:45.292128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:45.292093 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd"] Oct 30 06:21:45.293297 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:21:45.293265 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649f0a68_7af7_49e6_a6d0_39afce89434a.slice/crio-9fafef055183503c95444ba71dbd8e252b21b8b84ad9e80f521790aa49e6de89 WatchSource:0}: Error finding container 9fafef055183503c95444ba71dbd8e252b21b8b84ad9e80f521790aa49e6de89: Status 404 returned error can't find the container with id 9fafef055183503c95444ba71dbd8e252b21b8b84ad9e80f521790aa49e6de89 Oct 30 06:21:45.397293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:45.397255 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" event={"ID":"649f0a68-7af7-49e6-a6d0-39afce89434a","Type":"ContainerStarted","Data":"9fafef055183503c95444ba71dbd8e252b21b8b84ad9e80f521790aa49e6de89"} Oct 30 06:21:45.569113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:45.569084 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ef84b6f-f535-4625-a871-4bc7597e373b" path="/var/lib/kubelet/pods/4ef84b6f-f535-4625-a871-4bc7597e373b/volumes" Oct 30 06:21:45.569612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:45.569598 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7566c8e-9567-4891-8e5c-cdc1221ea5df" path="/var/lib/kubelet/pods/b7566c8e-9567-4891-8e5c-cdc1221ea5df/volumes" Oct 30 06:21:46.410108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:46.409996 5565 generic.go:356] "Generic (PLEG): container finished" podID="649f0a68-7af7-49e6-a6d0-39afce89434a" containerID="b1e34850a8ef2b7b8437c1b442713e2147962ee629d382215060f3221ac82fd3" exitCode=0 Oct 30 06:21:46.410108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:46.410091 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" event={"ID":"649f0a68-7af7-49e6-a6d0-39afce89434a","Type":"ContainerDied","Data":"b1e34850a8ef2b7b8437c1b442713e2147962ee629d382215060f3221ac82fd3"} Oct 30 06:21:47.548571 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:21:47.546969 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:21:47.595253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.595223 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:47.640328 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.640295 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca\") pod \"649f0a68-7af7-49e6-a6d0-39afce89434a\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " Oct 30 06:21:47.640484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.640403 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca\") pod \"649f0a68-7af7-49e6-a6d0-39afce89434a\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " Oct 30 06:21:47.640484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.640465 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj\" (UniqueName: \"kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj\") pod \"649f0a68-7af7-49e6-a6d0-39afce89434a\" (UID: \"649f0a68-7af7-49e6-a6d0-39afce89434a\") " Oct 30 06:21:47.640762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.640738 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca" pod "649f0a68-7af7-49e6-a6d0-39afce89434a" (UID: "649f0a68-7af7-49e6-a6d0-39afce89434a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:47.644797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.644466 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca" pod "649f0a68-7af7-49e6-a6d0-39afce89434a" (UID: "649f0a68-7af7-49e6-a6d0-39afce89434a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:47.644797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.644597 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:47.645127 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.645104 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj" pod "649f0a68-7af7-49e6-a6d0-39afce89434a" (UID: "649f0a68-7af7-49e6-a6d0-39afce89434a"). InnerVolumeSpecName "kube-api-access-nt9kj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:47.746183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.746102 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/649f0a68-7af7-49e6-a6d0-39afce89434a-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:47.746183 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:47.746133 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nt9kj\" (UniqueName: \"kubernetes.io/projected/649f0a68-7af7-49e6-a6d0-39afce89434a-kube-api-access-nt9kj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:48.427412 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:48.427375 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" Oct 30 06:21:48.427412 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:48.427399 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd" event={"ID":"649f0a68-7af7-49e6-a6d0-39afce89434a","Type":"ContainerDied","Data":"9fafef055183503c95444ba71dbd8e252b21b8b84ad9e80f521790aa49e6de89"} Oct 30 06:21:48.427743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:48.427450 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fafef055183503c95444ba71dbd8e252b21b8b84ad9e80f521790aa49e6de89" Oct 30 06:21:49.453064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.453023 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d"] Oct 30 06:21:49.525541 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.525482 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq"] Oct 30 06:21:49.526179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.526160 5565 cpu_manager.go:414] "containerMap: removing container" podUID="649f0a68-7af7-49e6-a6d0-39afce89434a" containerName="revoke-keys" Oct 30 06:21:49.526230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.526181 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649f0a68-7af7-49e6-a6d0-39afce89434a" containerName="revoke-keys" Oct 30 06:21:49.526309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.526299 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="649f0a68-7af7-49e6-a6d0-39afce89434a" containerName="revoke-keys" Oct 30 06:21:49.529960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.529936 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.546809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.546773 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:21:49.546809 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.546785 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:21:49.547370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.547349 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq"] Oct 30 06:21:49.605717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.605674 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.605917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.605874 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlsf5\" (UniqueName: \"kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.605917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.605915 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.707099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.706995 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.707295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.707163 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlsf5\" (UniqueName: \"kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.707295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.707218 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.707944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.707916 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.709495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.709472 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.737187 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.737146 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlsf5\" (UniqueName: \"kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:49.844906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:49.844870 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:50.152571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:50.152546 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq"] Oct 30 06:21:50.155080 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:21:50.155049 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc0812c_6f8b_4a79_aabf_79a8255ecef4.slice/crio-0db86a88849ecdf5152da0f118c64116e17948aaf59172fb3558d40f673c0d53 WatchSource:0}: Error finding container 0db86a88849ecdf5152da0f118c64116e17948aaf59172fb3558d40f673c0d53: Status 404 returned error can't find the container with id 0db86a88849ecdf5152da0f118c64116e17948aaf59172fb3558d40f673c0d53 Oct 30 06:21:50.445742 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:50.445652 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" event={"ID":"cdc0812c-6f8b-4a79-aabf-79a8255ecef4","Type":"ContainerStarted","Data":"0db86a88849ecdf5152da0f118c64116e17948aaf59172fb3558d40f673c0d53"} Oct 30 06:21:51.454282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:51.454182 5565 generic.go:356] "Generic (PLEG): container finished" podID="cdc0812c-6f8b-4a79-aabf-79a8255ecef4" containerID="cfeab602097c6477ee97bfd701d3e066ab5bef6fece5249d69ef9ecbbce13bd2" exitCode=0 Oct 30 06:21:51.454282 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:51.454263 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" event={"ID":"cdc0812c-6f8b-4a79-aabf-79a8255ecef4","Type":"ContainerDied","Data":"cfeab602097c6477ee97bfd701d3e066ab5bef6fece5249d69ef9ecbbce13bd2"} Oct 30 06:21:52.609790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.609764 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:52.746143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.746065 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca\") pod \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " Oct 30 06:21:52.746143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.746141 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5\" (UniqueName: \"kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5\") pod \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " Oct 30 06:21:52.746407 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.746301 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca\") pod \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\" (UID: \"cdc0812c-6f8b-4a79-aabf-79a8255ecef4\") " Oct 30 06:21:52.746551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.746504 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca" pod "cdc0812c-6f8b-4a79-aabf-79a8255ecef4" (UID: "cdc0812c-6f8b-4a79-aabf-79a8255ecef4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:21:52.746920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.746892 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:52.749000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.748971 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca" pod "cdc0812c-6f8b-4a79-aabf-79a8255ecef4" (UID: "cdc0812c-6f8b-4a79-aabf-79a8255ecef4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:21:52.749000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.748987 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5" pod "cdc0812c-6f8b-4a79-aabf-79a8255ecef4" (UID: "cdc0812c-6f8b-4a79-aabf-79a8255ecef4"). InnerVolumeSpecName "kube-api-access-dlsf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:21:52.847988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.847963 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlsf5\" (UniqueName: \"kubernetes.io/projected/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-kube-api-access-dlsf5\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:52.847988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:52.847987 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc0812c-6f8b-4a79-aabf-79a8255ecef4-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:21:53.469606 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:53.469582 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" Oct 30 06:21:53.469783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:53.469577 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq" event={"ID":"cdc0812c-6f8b-4a79-aabf-79a8255ecef4","Type":"ContainerDied","Data":"0db86a88849ecdf5152da0f118c64116e17948aaf59172fb3558d40f673c0d53"} Oct 30 06:21:53.469783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:53.469702 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0db86a88849ecdf5152da0f118c64116e17948aaf59172fb3558d40f673c0d53" Oct 30 06:21:55.045215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:55.045179 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2"] Oct 30 06:21:58.078014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:58.077979 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:21:59.871347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:21:59.871306 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz"] Oct 30 06:22:00.484990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:00.484926 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" podUID="6f2c6831-2df4-4b85-a226-51627753114d" containerName="payload-pre-processing" containerID="cri-o://c5e731b9f2238af08360c0aaddd5a71be950f320e9dbc12db61c2ebbbcf6346b" gracePeriod=25 Oct 30 06:22:04.478479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.478410 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" containerID="cri-o://be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b" gracePeriod=45 Oct 30 06:22:04.837076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.837050 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 06:22:04.872250 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.872216 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" containerName="payload-processing" containerID="cri-o://36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60" gracePeriod=25 Oct 30 06:22:04.926697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.926662 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home\") pod \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " Oct 30 06:22:04.926906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.926736 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm\") pod \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " Oct 30 06:22:04.926906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.926779 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir\") pod \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " Oct 30 06:22:04.926906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.926867 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home" pod "8aea6cb5-f76d-49ca-891b-2779f4f3689a" (UID: "8aea6cb5-f76d-49ca-891b-2779f4f3689a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.927105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.926982 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache\") pod \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " Oct 30 06:22:04.927105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.927060 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c\" (UniqueName: \"kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c\") pod \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " Oct 30 06:22:04.927105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.927092 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs\") pod \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\" (UID: \"8aea6cb5-f76d-49ca-891b-2779f4f3689a\") " Oct 30 06:22:04.927284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.927136 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir" pod "8aea6cb5-f76d-49ca-891b-2779f4f3689a" (UID: "8aea6cb5-f76d-49ca-891b-2779f4f3689a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.927676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.927651 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache" pod "8aea6cb5-f76d-49ca-891b-2779f4f3689a" (UID: "8aea6cb5-f76d-49ca-891b-2779f4f3689a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:04.927839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.927768 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:04.927985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.927852 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:04.929763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.929717 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c" pod "8aea6cb5-f76d-49ca-891b-2779f4f3689a" (UID: "8aea6cb5-f76d-49ca-891b-2779f4f3689a"). InnerVolumeSpecName "kube-api-access-cf26c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:04.930409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.930370 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs" pod "8aea6cb5-f76d-49ca-891b-2779f4f3689a" (UID: "8aea6cb5-f76d-49ca-891b-2779f4f3689a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:04.931009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:04.930987 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm" pod "8aea6cb5-f76d-49ca-891b-2779f4f3689a" (UID: "8aea6cb5-f76d-49ca-891b-2779f4f3689a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:05.029010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.028979 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.029010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.029010 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8aea6cb5-f76d-49ca-891b-2779f4f3689a-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.029213 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.029025 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf26c\" (UniqueName: \"kubernetes.io/projected/8aea6cb5-f76d-49ca-891b-2779f4f3689a-kube-api-access-cf26c\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.029213 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.029036 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8aea6cb5-f76d-49ca-891b-2779f4f3689a-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:05.577928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.577888 5565 generic.go:356] "Generic (PLEG): container finished" podID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerID="be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b" exitCode=0 Oct 30 06:22:05.577928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.577932 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" Oct 30 06:22:05.578448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.577962 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" event={"ID":"8aea6cb5-f76d-49ca-891b-2779f4f3689a","Type":"ContainerDied","Data":"be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b"} Oct 30 06:22:05.578448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.578000 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d" event={"ID":"8aea6cb5-f76d-49ca-891b-2779f4f3689a","Type":"ContainerDied","Data":"5ad4cd13c7484d37f42579be77a328879d50f181ec41c39ad7f3d27062efa5e8"} Oct 30 06:22:05.578448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.578016 5565 scope.go:122] "RemoveContainer" containerID="be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b" Oct 30 06:22:05.589572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.589554 5565 scope.go:122] "RemoveContainer" containerID="be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b" Oct 30 06:22:05.589826 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:05.589805 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b\": container with ID starting with be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b not found: ID does not exist" containerID="be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b" Oct 30 06:22:05.589885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.589833 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b"} err="failed to get container status \"be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b\": rpc error: code = NotFound desc = could not find container \"be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b\": container with ID starting with be3e7a578178f47a70d0a2da88fe3cc4b0acfb2a4c7143f9962bcafd4216b88b not found: ID does not exist" Oct 30 06:22:05.635308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.635281 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d"] Oct 30 06:22:05.639473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:05.639449 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-a14091/simulator-w0-a14091-kserve-5dc675cd75-qdt4d"] Oct 30 06:22:07.568304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:07.568271 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" path="/var/lib/kubelet/pods/8aea6cb5-f76d-49ca-891b-2779f4f3689a/volumes" Oct 30 06:22:07.659341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:07.659311 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" podUID="6f2c6831-2df4-4b85-a226-51627753114d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:12.683756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:12.683705 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:17.533506 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:17.533466 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:22:17.659600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:17.659558 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" podUID="6f2c6831-2df4-4b85-a226-51627753114d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:18.553607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.553580 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p"] Oct 30 06:22:18.554262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.554246 5565 cpu_manager.go:414] "containerMap: removing container" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" Oct 30 06:22:18.554309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.554263 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" Oct 30 06:22:18.554309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.554287 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cdc0812c-6f8b-4a79-aabf-79a8255ecef4" containerName="revoke-keys" Oct 30 06:22:18.554309 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.554292 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc0812c-6f8b-4a79-aabf-79a8255ecef4" containerName="revoke-keys" Oct 30 06:22:18.554408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.554390 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aea6cb5-f76d-49ca-891b-2779f4f3689a" containerName="main" Oct 30 06:22:18.554451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.554409 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc0812c-6f8b-4a79-aabf-79a8255ecef4" containerName="revoke-keys" Oct 30 06:22:18.558732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.558712 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.577767 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.577738 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p"] Oct 30 06:22:18.588545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.588510 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:18.588655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.588622 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:22:18.708495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.708451 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.708685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.708642 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnndg\" (UniqueName: \"kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.708733 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.708691 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.810128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.810058 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.810263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.810163 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnndg\" (UniqueName: \"kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.810263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.810193 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.810804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.810782 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.812422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.812400 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.835422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.835389 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnndg\" (UniqueName: \"kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg\") pod \"maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:18.871792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:18.871772 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:19.177366 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:19.177338 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p"] Oct 30 06:22:19.179710 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:22:19.179683 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9235d4bf_58b3_4a2b_8e86_b915affdb1d8.slice/crio-a02e95b0a9a6a944646ce65632c1bd301663199eba21b873af805130767afceb WatchSource:0}: Error finding container a02e95b0a9a6a944646ce65632c1bd301663199eba21b873af805130767afceb: Status 404 returned error can't find the container with id a02e95b0a9a6a944646ce65632c1bd301663199eba21b873af805130767afceb Oct 30 06:22:19.750485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:19.750376 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" event={"ID":"9235d4bf-58b3-4a2b-8e86-b915affdb1d8","Type":"ContainerStarted","Data":"6ef621e34cc43a472edf239642e529f431c36d59d597d346fdf5c06f4592d7be"} Oct 30 06:22:19.751102 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:19.751067 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" event={"ID":"9235d4bf-58b3-4a2b-8e86-b915affdb1d8","Type":"ContainerStarted","Data":"a02e95b0a9a6a944646ce65632c1bd301663199eba21b873af805130767afceb"} Oct 30 06:22:19.808010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:19.807958 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" podStartSLOduration=1.8079441040000002 podStartE2EDuration="1.807944104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:19.807704089 +0000 UTC m=+3046.383391145" watchObservedRunningTime="2026-10-30 06:22:19.807944104 +0000 UTC m=+3046.383631158" Oct 30 06:22:20.309857 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.309777 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq"] Oct 30 06:22:20.323484 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.323458 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-080391-mprhq"] Oct 30 06:22:20.759142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.759109 5565 generic.go:356] "Generic (PLEG): container finished" podID="6f2c6831-2df4-4b85-a226-51627753114d" containerID="c5e731b9f2238af08360c0aaddd5a71be950f320e9dbc12db61c2ebbbcf6346b" exitCode=0 Oct 30 06:22:20.759713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.759183 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" event={"ID":"6f2c6831-2df4-4b85-a226-51627753114d","Type":"ContainerDied","Data":"c5e731b9f2238af08360c0aaddd5a71be950f320e9dbc12db61c2ebbbcf6346b"} Oct 30 06:22:20.759713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.759259 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" event={"ID":"6f2c6831-2df4-4b85-a226-51627753114d","Type":"ContainerDied","Data":"ae939468ade3fba64885cb0ff2f386d600ae16a68f0d1c1d5b26dc2b6d323f58"} Oct 30 06:22:20.759713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.759286 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ae939468ade3fba64885cb0ff2f386d600ae16a68f0d1c1d5b26dc2b6d323f58" Oct 30 06:22:20.761115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.761078 5565 generic.go:356] "Generic (PLEG): container finished" podID="9235d4bf-58b3-4a2b-8e86-b915affdb1d8" containerID="6ef621e34cc43a472edf239642e529f431c36d59d597d346fdf5c06f4592d7be" exitCode=0 Oct 30 06:22:20.761231 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.761133 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" event={"ID":"9235d4bf-58b3-4a2b-8e86-b915affdb1d8","Type":"ContainerDied","Data":"6ef621e34cc43a472edf239642e529f431c36d59d597d346fdf5c06f4592d7be"} Oct 30 06:22:20.779715 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.779695 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 06:22:20.858350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.858279 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume\") pod \"6f2c6831-2df4-4b85-a226-51627753114d\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " Oct 30 06:22:20.858350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.858338 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc\" (UniqueName: \"kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc\") pod \"6f2c6831-2df4-4b85-a226-51627753114d\" (UID: \"6f2c6831-2df4-4b85-a226-51627753114d\") " Oct 30 06:22:20.858756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.858730 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume" pod "6f2c6831-2df4-4b85-a226-51627753114d" (UID: "6f2c6831-2df4-4b85-a226-51627753114d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:20.858956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.858941 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f2c6831-2df4-4b85-a226-51627753114d-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:20.860803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.860777 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc" pod "6f2c6831-2df4-4b85-a226-51627753114d" (UID: "6f2c6831-2df4-4b85-a226-51627753114d"). InnerVolumeSpecName "kube-api-access-d8qgc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:20.960325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:20.960289 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8qgc\" (UniqueName: \"kubernetes.io/projected/6f2c6831-2df4-4b85-a226-51627753114d-kube-api-access-d8qgc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:21.568351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:21.568310 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc0812c-6f8b-4a79-aabf-79a8255ecef4" path="/var/lib/kubelet/pods/cdc0812c-6f8b-4a79-aabf-79a8255ecef4/volumes" Oct 30 06:22:21.768032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:21.767999 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2" Oct 30 06:22:21.827144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:21.827114 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2"] Oct 30 06:22:21.841478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:21.841447 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-a14091-7567dc9bb7-wh5b2"] Oct 30 06:22:21.950993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:21.950965 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:22.074092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.074011 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca\") pod \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " Oct 30 06:22:22.074092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.074067 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg\" (UniqueName: \"kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg\") pod \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " Oct 30 06:22:22.074278 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.074114 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca\") pod \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\" (UID: \"9235d4bf-58b3-4a2b-8e86-b915affdb1d8\") " Oct 30 06:22:22.074604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.074573 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca" pod "9235d4bf-58b3-4a2b-8e86-b915affdb1d8" (UID: "9235d4bf-58b3-4a2b-8e86-b915affdb1d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:22.076200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.076174 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca" pod "9235d4bf-58b3-4a2b-8e86-b915affdb1d8" (UID: "9235d4bf-58b3-4a2b-8e86-b915affdb1d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:22.076325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.076232 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg" pod "9235d4bf-58b3-4a2b-8e86-b915affdb1d8" (UID: "9235d4bf-58b3-4a2b-8e86-b915affdb1d8"). InnerVolumeSpecName "kube-api-access-gnndg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:22.112254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.112230 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6"] Oct 30 06:22:22.112477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.112439 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" podUID="68f9db15-ee0f-4486-b455-d63e690d9e78" containerName="payload-processing" containerID="cri-o://b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0" gracePeriod=30 Oct 30 06:22:22.175509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.175481 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.175509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.175511 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnndg\" (UniqueName: \"kubernetes.io/projected/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-kube-api-access-gnndg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.175718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.175559 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9235d4bf-58b3-4a2b-8e86-b915affdb1d8-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.366412 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.366389 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:22:22.494557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.480142 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume\") pod \"68f9db15-ee0f-4486-b455-d63e690d9e78\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " Oct 30 06:22:22.507557 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.506016 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh\" (UniqueName: \"kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh\") pod \"68f9db15-ee0f-4486-b455-d63e690d9e78\" (UID: \"68f9db15-ee0f-4486-b455-d63e690d9e78\") " Oct 30 06:22:22.516594 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.507958 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume" pod "68f9db15-ee0f-4486-b455-d63e690d9e78" (UID: "68f9db15-ee0f-4486-b455-d63e690d9e78"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:22.546805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.546762 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh" pod "68f9db15-ee0f-4486-b455-d63e690d9e78" (UID: "68f9db15-ee0f-4486-b455-d63e690d9e78"). InnerVolumeSpecName "kube-api-access-k98bh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:22.609036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.608619 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k98bh\" (UniqueName: \"kubernetes.io/projected/68f9db15-ee0f-4486-b455-d63e690d9e78-kube-api-access-k98bh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.609036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.608676 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/68f9db15-ee0f-4486-b455-d63e690d9e78-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:22.683310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.683269 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:22.797950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.797909 5565 generic.go:356] "Generic (PLEG): container finished" podID="68f9db15-ee0f-4486-b455-d63e690d9e78" containerID="b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0" exitCode=0 Oct 30 06:22:22.798653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.797966 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" Oct 30 06:22:22.798653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.798030 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" event={"ID":"68f9db15-ee0f-4486-b455-d63e690d9e78","Type":"ContainerDied","Data":"b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0"} Oct 30 06:22:22.798653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.798114 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6" event={"ID":"68f9db15-ee0f-4486-b455-d63e690d9e78","Type":"ContainerDied","Data":"43724d03642a139724e738cf416110c0d4b6bfc5b37fafbff1e046e4915f0de0"} Oct 30 06:22:22.798653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.798145 5565 scope.go:122] "RemoveContainer" containerID="b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0" Oct 30 06:22:22.801596 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.801570 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" Oct 30 06:22:22.801706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.801570 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p" event={"ID":"9235d4bf-58b3-4a2b-8e86-b915affdb1d8","Type":"ContainerDied","Data":"a02e95b0a9a6a944646ce65632c1bd301663199eba21b873af805130767afceb"} Oct 30 06:22:22.801706 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.801681 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a02e95b0a9a6a944646ce65632c1bd301663199eba21b873af805130767afceb" Oct 30 06:22:22.815386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.815364 5565 scope.go:122] "RemoveContainer" containerID="b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0" Oct 30 06:22:22.815666 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:22.815646 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0\": container with ID starting with b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0 not found: ID does not exist" containerID="b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0" Oct 30 06:22:22.815734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.815673 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0"} err="failed to get container status \"b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0\": rpc error: code = NotFound desc = could not find container \"b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0\": container with ID starting with b9829cfe8e3c3d4fe3789b944daddac0eb5208d06fe5b323c957a6d52844e1a0 not found: ID does not exist" Oct 30 06:22:22.858368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.858343 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6"] Oct 30 06:22:22.864824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:22.864758 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-07b278-76f67fd7f8-s6lm6"] Oct 30 06:22:23.569182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:23.569150 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68f9db15-ee0f-4486-b455-d63e690d9e78" path="/var/lib/kubelet/pods/68f9db15-ee0f-4486-b455-d63e690d9e78/volumes" Oct 30 06:22:23.569578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:23.569563 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f2c6831-2df4-4b85-a226-51627753114d" path="/var/lib/kubelet/pods/6f2c6831-2df4-4b85-a226-51627753114d/volumes" Oct 30 06:22:25.153772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.153749 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:22:25.240379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.240337 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx\" (UniqueName: \"kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx\") pod \"bd12a404-1cde-4e6e-96e0-38035239d979\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " Oct 30 06:22:25.240574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.240405 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume\") pod \"bd12a404-1cde-4e6e-96e0-38035239d979\" (UID: \"bd12a404-1cde-4e6e-96e0-38035239d979\") " Oct 30 06:22:25.240867 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.240827 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume" pod "bd12a404-1cde-4e6e-96e0-38035239d979" (UID: "bd12a404-1cde-4e6e-96e0-38035239d979"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:25.241058 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.241038 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bd12a404-1cde-4e6e-96e0-38035239d979-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.242814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.242789 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx" pod "bd12a404-1cde-4e6e-96e0-38035239d979" (UID: "bd12a404-1cde-4e6e-96e0-38035239d979"). InnerVolumeSpecName "kube-api-access-rmnnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:25.342060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.342029 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmnnx\" (UniqueName: \"kubernetes.io/projected/bd12a404-1cde-4e6e-96e0-38035239d979-kube-api-access-rmnnx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:25.826676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.826592 5565 generic.go:356] "Generic (PLEG): container finished" podID="bd12a404-1cde-4e6e-96e0-38035239d979" containerID="36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60" exitCode=0 Oct 30 06:22:25.826676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.826636 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" Oct 30 06:22:25.826911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.826683 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" event={"ID":"bd12a404-1cde-4e6e-96e0-38035239d979","Type":"ContainerDied","Data":"36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60"} Oct 30 06:22:25.826911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.826744 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz" event={"ID":"bd12a404-1cde-4e6e-96e0-38035239d979","Type":"ContainerDied","Data":"7e8983e5256e4f194a80688eacecde40b6f01d4e844a7148a74ed184aa5c19ab"} Oct 30 06:22:25.826911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.826774 5565 scope.go:122] "RemoveContainer" containerID="36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60" Oct 30 06:22:25.837073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.837053 5565 scope.go:122] "RemoveContainer" containerID="36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60" Oct 30 06:22:25.837336 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:25.837315 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60\": container with ID starting with 36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60 not found: ID does not exist" containerID="36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60" Oct 30 06:22:25.837390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.837344 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60"} err="failed to get container status \"36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60\": rpc error: code = NotFound desc = could not find container \"36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60\": container with ID starting with 36a6aa87339a8a8695db879fbcb9655c2e661d5bccfa0510dfb055d46a41ff60 not found: ID does not exist" Oct 30 06:22:25.888702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.886402 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz"] Oct 30 06:22:25.897909 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:25.897884 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-080391-7bb8f4c5cb-kbllz"] Oct 30 06:22:27.568951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:27.568917 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" path="/var/lib/kubelet/pods/bd12a404-1cde-4e6e-96e0-38035239d979/volumes" Oct 30 06:22:28.259294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:28.259259 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66"] Oct 30 06:22:28.259782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:28.259748 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" containerID="cri-o://4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a" gracePeriod=30 Oct 30 06:22:30.350029 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.349996 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8"] Oct 30 06:22:30.350632 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.350247 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" containerID="cri-o://6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9" gracePeriod=30 Oct 30 06:22:30.877990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.877954 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl"] Oct 30 06:22:30.878583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878566 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6f2c6831-2df4-4b85-a226-51627753114d" containerName="payload-pre-processing" Oct 30 06:22:30.878583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878582 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f2c6831-2df4-4b85-a226-51627753114d" containerName="payload-pre-processing" Oct 30 06:22:30.878726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878608 5565 cpu_manager.go:414] "containerMap: removing container" podUID="68f9db15-ee0f-4486-b455-d63e690d9e78" containerName="payload-processing" Oct 30 06:22:30.878726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878613 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68f9db15-ee0f-4486-b455-d63e690d9e78" containerName="payload-processing" Oct 30 06:22:30.878726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878640 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9235d4bf-58b3-4a2b-8e86-b915affdb1d8" containerName="revoke-keys" Oct 30 06:22:30.878726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878645 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9235d4bf-58b3-4a2b-8e86-b915affdb1d8" containerName="revoke-keys" Oct 30 06:22:30.878726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878655 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" containerName="payload-processing" Oct 30 06:22:30.878726 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878660 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" containerName="payload-processing" Oct 30 06:22:30.878983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878744 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9235d4bf-58b3-4a2b-8e86-b915affdb1d8" containerName="revoke-keys" Oct 30 06:22:30.878983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878755 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd12a404-1cde-4e6e-96e0-38035239d979" containerName="payload-processing" Oct 30 06:22:30.878983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878764 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f2c6831-2df4-4b85-a226-51627753114d" containerName="payload-pre-processing" Oct 30 06:22:30.878983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.878773 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="68f9db15-ee0f-4486-b455-d63e690d9e78" containerName="payload-processing" Oct 30 06:22:30.883940 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.883919 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:30.897424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.897387 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl"] Oct 30 06:22:30.922569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.920123 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:30.922569 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.920242 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:22:30.931887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.931850 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5zbs\" (UniqueName: \"kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:30.932046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.931894 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:30.932046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:30.931966 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.033016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.032978 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5zbs\" (UniqueName: \"kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.033228 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.033022 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.033228 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.033062 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.033737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.033701 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.035448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.035423 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.058030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.057994 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5zbs\" (UniqueName: \"kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs\") pod \"maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.200242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.200149 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:31.515051 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.515020 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl"] Oct 30 06:22:31.516316 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:22:31.516287 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf861b9e_af63_4f3e_9858_71c398e67cdd.slice/crio-03f1d66e404905cc51767cced9dbf8baf18c5771b9342847dd98f247db2c9ac5 WatchSource:0}: Error finding container 03f1d66e404905cc51767cced9dbf8baf18c5771b9342847dd98f247db2c9ac5: Status 404 returned error can't find the container with id 03f1d66e404905cc51767cced9dbf8baf18c5771b9342847dd98f247db2c9ac5 Oct 30 06:22:31.877387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.877353 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" event={"ID":"cf861b9e-af63-4f3e-9858-71c398e67cdd","Type":"ContainerStarted","Data":"3a0eb26f69623b0c18a285e6e107379a6cd70d2b22c1988295c6c2d8ea2d0a4b"} Oct 30 06:22:31.877868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.877409 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" event={"ID":"cf861b9e-af63-4f3e-9858-71c398e67cdd","Type":"ContainerStarted","Data":"03f1d66e404905cc51767cced9dbf8baf18c5771b9342847dd98f247db2c9ac5"} Oct 30 06:22:31.921652 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.921618 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br"] Oct 30 06:22:31.921933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.921879 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" podUID="9d0a8f40-9241-49f3-b904-7e0df242f7f8" containerName="payload-pre-processing" containerID="cri-o://434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565" gracePeriod=30 Oct 30 06:22:31.925304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:31.925245 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" podStartSLOduration=1.925229732 podStartE2EDuration="1.925229732s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:22:31.921163086 +0000 UTC m=+3058.496850141" watchObservedRunningTime="2026-10-30 06:22:31.925229732 +0000 UTC m=+3058.500916784" Oct 30 06:22:32.192646 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.192613 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:22:32.261267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.260144 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume\") pod \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " Oct 30 06:22:32.261267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.260202 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr\") pod \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\" (UID: \"9d0a8f40-9241-49f3-b904-7e0df242f7f8\") " Oct 30 06:22:32.261267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.260888 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume" pod "9d0a8f40-9241-49f3-b904-7e0df242f7f8" (UID: "9d0a8f40-9241-49f3-b904-7e0df242f7f8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:32.295553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.284743 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr" pod "9d0a8f40-9241-49f3-b904-7e0df242f7f8" (UID: "9d0a8f40-9241-49f3-b904-7e0df242f7f8"). InnerVolumeSpecName "kube-api-access-cwnxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:32.363114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.363074 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9d0a8f40-9241-49f3-b904-7e0df242f7f8-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.363114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.363114 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwnxr\" (UniqueName: \"kubernetes.io/projected/9d0a8f40-9241-49f3-b904-7e0df242f7f8-kube-api-access-cwnxr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:32.886139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.886106 5565 generic.go:356] "Generic (PLEG): container finished" podID="9d0a8f40-9241-49f3-b904-7e0df242f7f8" containerID="434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565" exitCode=0 Oct 30 06:22:32.886647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.886159 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" Oct 30 06:22:32.886647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.886185 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" event={"ID":"9d0a8f40-9241-49f3-b904-7e0df242f7f8","Type":"ContainerDied","Data":"434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565"} Oct 30 06:22:32.886647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.886235 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br" event={"ID":"9d0a8f40-9241-49f3-b904-7e0df242f7f8","Type":"ContainerDied","Data":"13365c176aac6c2bc394a68a5213b080b41132f0eb2f52fa1013175c32f6a95a"} Oct 30 06:22:32.886647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.886257 5565 scope.go:122] "RemoveContainer" containerID="434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565" Oct 30 06:22:32.887931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.887906 5565 generic.go:356] "Generic (PLEG): container finished" podID="cf861b9e-af63-4f3e-9858-71c398e67cdd" containerID="3a0eb26f69623b0c18a285e6e107379a6cd70d2b22c1988295c6c2d8ea2d0a4b" exitCode=0 Oct 30 06:22:32.888024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.887980 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" event={"ID":"cf861b9e-af63-4f3e-9858-71c398e67cdd","Type":"ContainerDied","Data":"3a0eb26f69623b0c18a285e6e107379a6cd70d2b22c1988295c6c2d8ea2d0a4b"} Oct 30 06:22:32.898157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.898137 5565 scope.go:122] "RemoveContainer" containerID="434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565" Oct 30 06:22:32.898471 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:32.898452 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565\": container with ID starting with 434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565 not found: ID does not exist" containerID="434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565" Oct 30 06:22:32.898555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.898482 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565"} err="failed to get container status \"434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565\": rpc error: code = NotFound desc = could not find container \"434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565\": container with ID starting with 434fd69a4e1521e4752b890ee795c664d441a5f7e6c43c85c526c4ef04a2b565 not found: ID does not exist" Oct 30 06:22:32.988769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.988731 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br"] Oct 30 06:22:32.991507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:32.991482 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a8b892-77f8bdbdf5-mr8br"] Oct 30 06:22:33.037994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.037950 5565 patch_prober.go:29] interesting pod/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:33.038157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.038016 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:33.535625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.535566 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:22:33.568846 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.568817 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d0a8f40-9241-49f3-b904-7e0df242f7f8" path="/var/lib/kubelet/pods/9d0a8f40-9241-49f3-b904-7e0df242f7f8/volumes" Oct 30 06:22:33.579016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.578990 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579035 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579135 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579227 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579255 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.579293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579273 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579318 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579363 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579406 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579456 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.579639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579494 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data\") pod \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\" (UID: \"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea\") " Oct 30 06:22:33.579639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579509 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.579639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579515 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:33.579913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.579889 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.580083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.580065 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.580168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.580088 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.580168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.580105 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.580168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.580118 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.580168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.580133 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.581197 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.581171 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:33.581640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.581619 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:33.581733 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.581651 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "kube-api-access-5n8c7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:33.581849 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.581831 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo" pod "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" (UID: "f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:33.680987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.680959 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.680987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.680986 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.681179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.680996 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n8c7\" (UniqueName: \"kubernetes.io/projected/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-kube-api-access-5n8c7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.681179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.681005 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:33.899487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.899397 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerID="4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a" exitCode=0 Oct 30 06:22:33.899487 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.899444 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" Oct 30 06:22:33.899996 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.899479 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" event={"ID":"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea","Type":"ContainerDied","Data":"4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a"} Oct 30 06:22:33.899996 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.899546 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66" event={"ID":"f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea","Type":"ContainerDied","Data":"30aee0f9256f6969a319e0c2beefa4bfa386628b4970d68a7e6a1ba6952af1bd"} Oct 30 06:22:33.899996 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.899576 5565 scope.go:122] "RemoveContainer" containerID="4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a" Oct 30 06:22:33.911090 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.911072 5565 scope.go:122] "RemoveContainer" containerID="31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472" Oct 30 06:22:33.922219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.922197 5565 scope.go:122] "RemoveContainer" containerID="4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a" Oct 30 06:22:33.922574 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:33.922521 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a\": container with ID starting with 4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a not found: ID does not exist" containerID="4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a" Oct 30 06:22:33.922677 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.922585 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a"} err="failed to get container status \"4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a\": rpc error: code = NotFound desc = could not find container \"4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a\": container with ID starting with 4c67389e18c86a0cd9b3e2216fc7cfdd4df980929aa920bfb170bcbd8002272a not found: ID does not exist" Oct 30 06:22:33.922677 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.922609 5565 scope.go:122] "RemoveContainer" containerID="31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472" Oct 30 06:22:33.922922 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:33.922895 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472\": container with ID starting with 31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472 not found: ID does not exist" containerID="31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472" Oct 30 06:22:33.923029 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.922928 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472"} err="failed to get container status \"31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472\": rpc error: code = NotFound desc = could not find container \"31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472\": container with ID starting with 31e7d4b4cdb0a2098cf11e91b3c7dea7f434cfe8c6a0c03caadfdbd4f89cf472 not found: ID does not exist" Oct 30 06:22:33.991107 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.991074 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66"] Oct 30 06:22:33.995334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:33.995308 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-080391-openshift-default-84cbf88d99-kst66"] Oct 30 06:22:34.028377 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.028355 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:34.085494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.085463 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca\") pod \"cf861b9e-af63-4f3e-9858-71c398e67cdd\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " Oct 30 06:22:34.085687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.085594 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs\" (UniqueName: \"kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs\") pod \"cf861b9e-af63-4f3e-9858-71c398e67cdd\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " Oct 30 06:22:34.085687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.085633 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca\") pod \"cf861b9e-af63-4f3e-9858-71c398e67cdd\" (UID: \"cf861b9e-af63-4f3e-9858-71c398e67cdd\") " Oct 30 06:22:34.086020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.085994 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca" pod "cf861b9e-af63-4f3e-9858-71c398e67cdd" (UID: "cf861b9e-af63-4f3e-9858-71c398e67cdd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:34.086255 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.086230 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:34.087636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.087610 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca" pod "cf861b9e-af63-4f3e-9858-71c398e67cdd" (UID: "cf861b9e-af63-4f3e-9858-71c398e67cdd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:34.087957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.087930 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs" pod "cf861b9e-af63-4f3e-9858-71c398e67cdd" (UID: "cf861b9e-af63-4f3e-9858-71c398e67cdd"). InnerVolumeSpecName "kube-api-access-x5zbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:34.187669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.187607 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5zbs\" (UniqueName: \"kubernetes.io/projected/cf861b9e-af63-4f3e-9858-71c398e67cdd-kube-api-access-x5zbs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:34.187669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.187633 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf861b9e-af63-4f3e-9858-71c398e67cdd-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:34.909331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.909303 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" Oct 30 06:22:34.909331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.909318 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl" event={"ID":"cf861b9e-af63-4f3e-9858-71c398e67cdd","Type":"ContainerDied","Data":"03f1d66e404905cc51767cced9dbf8baf18c5771b9342847dd98f247db2c9ac5"} Oct 30 06:22:34.909824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:34.909367 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03f1d66e404905cc51767cced9dbf8baf18c5771b9342847dd98f247db2c9ac5" Oct 30 06:22:35.570672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.570641 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" path="/var/lib/kubelet/pods/f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea/volumes" Oct 30 06:22:35.628182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.628123 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:22:35.704890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.704862 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.704901 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705064 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705103 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705183 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:35.705267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705233 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705282 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705409 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705438 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705512 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo\") pod \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\" (UID: \"121ba31d-7c52-4d21-bac3-fc3848d2a8f4\") " Oct 30 06:22:35.705673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705649 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:35.705923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.705899 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:35.706355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.706319 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.706355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.706354 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.706578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.706368 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.706578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.706444 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:35.706578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.706561 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:35.707834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.707812 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "kube-api-access-qk2nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:35.708257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.708230 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:22:35.708589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.708567 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:35.708681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.708632 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy" pod "121ba31d-7c52-4d21-bac3-fc3848d2a8f4" (UID: "121ba31d-7c52-4d21-bac3-fc3848d2a8f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:22:35.808086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.808048 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.808086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.808082 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.808086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.808096 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.808379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.808107 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2nb\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-kube-api-access-qk2nb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.808379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.808116 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.808379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.808125 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/121ba31d-7c52-4d21-bac3-fc3848d2a8f4-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:35.919905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.919804 5565 generic.go:356] "Generic (PLEG): container finished" podID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerID="6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9" exitCode=0 Oct 30 06:22:35.919905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.919866 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" Oct 30 06:22:35.919905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.919896 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" event={"ID":"121ba31d-7c52-4d21-bac3-fc3848d2a8f4","Type":"ContainerDied","Data":"6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9"} Oct 30 06:22:35.920467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.919953 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8" event={"ID":"121ba31d-7c52-4d21-bac3-fc3848d2a8f4","Type":"ContainerDied","Data":"1ff2411f68ff89f3c5b96d2049d43709bed67d555ca3f1004afcb82f9e818332"} Oct 30 06:22:35.920467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.919979 5565 scope.go:122] "RemoveContainer" containerID="6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9" Oct 30 06:22:35.931427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.931406 5565 scope.go:122] "RemoveContainer" containerID="073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403" Oct 30 06:22:35.942780 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.942755 5565 scope.go:122] "RemoveContainer" containerID="6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9" Oct 30 06:22:35.943066 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:35.943045 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9\": container with ID starting with 6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9 not found: ID does not exist" containerID="6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9" Oct 30 06:22:35.943142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.943081 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9"} err="failed to get container status \"6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9\": rpc error: code = NotFound desc = could not find container \"6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9\": container with ID starting with 6d93e6b813a9f189b1079016f33559002b4b4e8e38e60a1c76c48d1ecb5913e9 not found: ID does not exist" Oct 30 06:22:35.943142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.943106 5565 scope.go:122] "RemoveContainer" containerID="073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403" Oct 30 06:22:35.943380 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:35.943361 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403\": container with ID starting with 073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403 not found: ID does not exist" containerID="073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403" Oct 30 06:22:35.943428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.943386 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403"} err="failed to get container status \"073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403\": rpc error: code = NotFound desc = could not find container \"073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403\": container with ID starting with 073a3c1d071cc43cb6133f8d536fb6ef09ca47e2b6e1c85676a24cf1b6bd6403 not found: ID does not exist" Oct 30 06:22:35.978575 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.978541 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8"] Oct 30 06:22:35.992154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:35.992121 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-07b278-openshift-default-7b789bb6dc-sn5h8"] Oct 30 06:22:37.568457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:37.568421 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" path="/var/lib/kubelet/pods/121ba31d-7c52-4d21-bac3-fc3848d2a8f4/volumes" Oct 30 06:22:39.184893 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.184858 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5"] Oct 30 06:22:39.218112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.218082 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd"] Oct 30 06:22:39.285897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.285868 5565 scope.go:122] "RemoveContainer" containerID="c5e731b9f2238af08360c0aaddd5a71be950f320e9dbc12db61c2ebbbcf6346b" Oct 30 06:22:39.299435 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.299401 5565 scope.go:122] "RemoveContainer" containerID="1fcfa13562e112254a168829a91706439168054bf1f0d01444025b44b9f0aafe" Oct 30 06:22:39.308672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.308654 5565 scope.go:122] "RemoveContainer" containerID="8abb0771aea53fd4bc4d7fafe23c3fca24bd60be21cc99889e2016f23fa1d050" Oct 30 06:22:39.317728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.317705 5565 scope.go:122] "RemoveContainer" containerID="6c8dde065c663a55ac701415d984b38358730e96b7e3353a65c4c7778176d51d" Oct 30 06:22:39.327185 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.327167 5565 scope.go:122] "RemoveContainer" containerID="18ee9a7b8d4b2fbadbd31de4229b7cf39850bd133824ab179ff1ee0b6f0fbabf" Oct 30 06:22:39.335643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.335628 5565 scope.go:122] "RemoveContainer" containerID="7dd899d8ba3360f8b79b52f8d0f51c8e28a21e9cf538647a74031a7aaca108b8" Oct 30 06:22:39.353324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:39.350692 5565 scope.go:122] "RemoveContainer" containerID="4b53a52da9e3e5aab117a459a76e4fe87e851a5c6bc66b8d971818bac544f2bf" Oct 30 06:22:40.046174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046140 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7"] Oct 30 06:22:40.046828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046812 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" Oct 30 06:22:40.046881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046829 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" Oct 30 06:22:40.046881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046840 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cf861b9e-af63-4f3e-9858-71c398e67cdd" containerName="revoke-keys" Oct 30 06:22:40.046881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046846 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf861b9e-af63-4f3e-9858-71c398e67cdd" containerName="revoke-keys" Oct 30 06:22:40.046881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046860 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9d0a8f40-9241-49f3-b904-7e0df242f7f8" containerName="payload-pre-processing" Oct 30 06:22:40.046881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046866 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d0a8f40-9241-49f3-b904-7e0df242f7f8" containerName="payload-pre-processing" Oct 30 06:22:40.046881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046878 5565 cpu_manager.go:414] "containerMap: removing container" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046885 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046912 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.046917 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.047005 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d0a8f40-9241-49f3-b904-7e0df242f7f8" containerName="payload-pre-processing" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.047014 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.047030 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.047038 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" Oct 30 06:22:40.047078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.047047 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf861b9e-af63-4f3e-9858-71c398e67cdd" containerName="revoke-keys" Oct 30 06:22:40.052062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.052037 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.070995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.070967 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7"] Oct 30 06:22:40.082828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.081651 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:22:40.082828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.081791 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:22:40.169970 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.169933 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktxgn\" (UniqueName: \"kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.170162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.170090 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.170265 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.170215 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.272380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.272341 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktxgn\" (UniqueName: \"kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.272854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.272416 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.272854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.272458 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.273329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.273303 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.274954 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.274933 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.298234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.298158 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktxgn\" (UniqueName: \"kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.366042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.366004 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:40.676914 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.676881 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7"] Oct 30 06:22:40.678726 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:22:40.678667 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef280ae8_0bc8_458a_bf67_b9a09ba41aec.slice/crio-22a8087937db81e4923b876fd10b158d425f1e27210e475ce209e067e1156cad WatchSource:0}: Error finding container 22a8087937db81e4923b876fd10b158d425f1e27210e475ce209e067e1156cad: Status 404 returned error can't find the container with id 22a8087937db81e4923b876fd10b158d425f1e27210e475ce209e067e1156cad Oct 30 06:22:40.792616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.792568 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:22:40.964229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:40.964130 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" event={"ID":"ef280ae8-0bc8-458a-bf67-b9a09ba41aec","Type":"ContainerStarted","Data":"22a8087937db81e4923b876fd10b158d425f1e27210e475ce209e067e1156cad"} Oct 30 06:22:41.972763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:41.972686 5565 generic.go:356] "Generic (PLEG): container finished" podID="ef280ae8-0bc8-458a-bf67-b9a09ba41aec" containerID="f81ff4c1212010b0f546d69da7eab545b7d52f3fc3491b12cee673da0ba4ae52" exitCode=0 Oct 30 06:22:41.973118 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:41.972767 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" event={"ID":"ef280ae8-0bc8-458a-bf67-b9a09ba41aec","Type":"ContainerDied","Data":"f81ff4c1212010b0f546d69da7eab545b7d52f3fc3491b12cee673da0ba4ae52"} Oct 30 06:22:43.127672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.127647 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:43.212617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.212585 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca\") pod \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " Oct 30 06:22:43.212818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.212651 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn\" (UniqueName: \"kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn\") pod \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " Oct 30 06:22:43.212818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.212754 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca\") pod \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\" (UID: \"ef280ae8-0bc8-458a-bf67-b9a09ba41aec\") " Oct 30 06:22:43.212998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.212973 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca" pod "ef280ae8-0bc8-458a-bf67-b9a09ba41aec" (UID: "ef280ae8-0bc8-458a-bf67-b9a09ba41aec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:43.213260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.213225 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:43.215086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.215056 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn" pod "ef280ae8-0bc8-458a-bf67-b9a09ba41aec" (UID: "ef280ae8-0bc8-458a-bf67-b9a09ba41aec"). InnerVolumeSpecName "kube-api-access-ktxgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:43.215523 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.215501 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca" pod "ef280ae8-0bc8-458a-bf67-b9a09ba41aec" (UID: "ef280ae8-0bc8-458a-bf67-b9a09ba41aec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:22:43.314052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.314025 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktxgn\" (UniqueName: \"kubernetes.io/projected/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-kube-api-access-ktxgn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:43.314052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.314049 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef280ae8-0bc8-458a-bf67-b9a09ba41aec-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:43.986588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.986477 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" Oct 30 06:22:43.986588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.986512 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7" event={"ID":"ef280ae8-0bc8-458a-bf67-b9a09ba41aec","Type":"ContainerDied","Data":"22a8087937db81e4923b876fd10b158d425f1e27210e475ce209e067e1156cad"} Oct 30 06:22:43.986588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:43.986578 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22a8087937db81e4923b876fd10b158d425f1e27210e475ce209e067e1156cad" Oct 30 06:22:44.185645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:44.185599 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerName="payload-processing" containerID="cri-o://5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b" gracePeriod=25 Oct 30 06:22:44.218785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:44.218755 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" podUID="820c0009-c7e3-4035-a4ba-733b27571362" containerName="payload-pre-processing" containerID="cri-o://93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e" gracePeriod=25 Oct 30 06:22:47.234865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.234780 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2"] Oct 30 06:22:47.235320 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.234981 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" podUID="b159bb4f-a575-4cae-8dc4-e6f5582fc629" containerName="payload-processing" containerID="cri-o://f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63" gracePeriod=30 Oct 30 06:22:47.269779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.269744 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962"] Oct 30 06:22:47.270237 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.270210 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" podUID="ca564ba7-731b-4d1d-b155-1dd05c326809" containerName="payload-pre-processing" containerID="cri-o://930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692" gracePeriod=30 Oct 30 06:22:47.565493 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:47.565454 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:22:47.586568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.570626 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:22:47.633200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.633164 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:22:47.663335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.663300 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc\" (UniqueName: \"kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc\") pod \"ca564ba7-731b-4d1d-b155-1dd05c326809\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " Oct 30 06:22:47.663335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.663348 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume\") pod \"ca564ba7-731b-4d1d-b155-1dd05c326809\" (UID: \"ca564ba7-731b-4d1d-b155-1dd05c326809\") " Oct 30 06:22:47.663648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.663408 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume\") pod \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " Oct 30 06:22:47.663648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.663484 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r\" (UniqueName: \"kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r\") pod \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\" (UID: \"b159bb4f-a575-4cae-8dc4-e6f5582fc629\") " Oct 30 06:22:47.664017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.663841 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume" pod "ca564ba7-731b-4d1d-b155-1dd05c326809" (UID: "ca564ba7-731b-4d1d-b155-1dd05c326809"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:47.664340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.664297 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume" pod "b159bb4f-a575-4cae-8dc4-e6f5582fc629" (UID: "b159bb4f-a575-4cae-8dc4-e6f5582fc629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:22:47.664443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.664391 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca564ba7-731b-4d1d-b155-1dd05c326809-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.666452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.666427 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r" pod "b159bb4f-a575-4cae-8dc4-e6f5582fc629" (UID: "b159bb4f-a575-4cae-8dc4-e6f5582fc629"). InnerVolumeSpecName "kube-api-access-8fb8r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:47.666806 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.666783 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc" pod "ca564ba7-731b-4d1d-b155-1dd05c326809" (UID: "ca564ba7-731b-4d1d-b155-1dd05c326809"). InnerVolumeSpecName "kube-api-access-zdcnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:22:47.765203 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.765168 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdcnc\" (UniqueName: \"kubernetes.io/projected/ca564ba7-731b-4d1d-b155-1dd05c326809-kube-api-access-zdcnc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.765203 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.765198 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b159bb4f-a575-4cae-8dc4-e6f5582fc629-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.765203 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.765207 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fb8r\" (UniqueName: \"kubernetes.io/projected/b159bb4f-a575-4cae-8dc4-e6f5582fc629-kube-api-access-8fb8r\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:22:47.768032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.767950 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p"] Oct 30 06:22:47.770420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:47.770397 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-7fdb95-rz49p"] Oct 30 06:22:48.014936 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.014896 5565 generic.go:356] "Generic (PLEG): container finished" podID="b159bb4f-a575-4cae-8dc4-e6f5582fc629" containerID="f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63" exitCode=0 Oct 30 06:22:48.014936 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.014940 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" Oct 30 06:22:48.015178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.014985 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" event={"ID":"b159bb4f-a575-4cae-8dc4-e6f5582fc629","Type":"ContainerDied","Data":"f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63"} Oct 30 06:22:48.015178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.015036 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2" event={"ID":"b159bb4f-a575-4cae-8dc4-e6f5582fc629","Type":"ContainerDied","Data":"61556bbd6586212321484f7d7054be19c7e7bb441ff33578223837c4e4a6b5e2"} Oct 30 06:22:48.015178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.015060 5565 scope.go:122] "RemoveContainer" containerID="f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63" Oct 30 06:22:48.016452 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.016434 5565 generic.go:356] "Generic (PLEG): container finished" podID="ca564ba7-731b-4d1d-b155-1dd05c326809" containerID="930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692" exitCode=0 Oct 30 06:22:48.016579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.016478 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" event={"ID":"ca564ba7-731b-4d1d-b155-1dd05c326809","Type":"ContainerDied","Data":"930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692"} Oct 30 06:22:48.016579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.016483 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" Oct 30 06:22:48.016579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.016513 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962" event={"ID":"ca564ba7-731b-4d1d-b155-1dd05c326809","Type":"ContainerDied","Data":"e2c08ad2ff2b724547c61ad6aee01fb8f952a76aa7902269822e07f8f6f3a882"} Oct 30 06:22:48.027030 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.027014 5565 scope.go:122] "RemoveContainer" containerID="f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63" Oct 30 06:22:48.027423 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:48.027375 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63\": container with ID starting with f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63 not found: ID does not exist" containerID="f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63" Oct 30 06:22:48.027536 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.027417 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63"} err="failed to get container status \"f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63\": rpc error: code = NotFound desc = could not find container \"f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63\": container with ID starting with f9e78f73303d74b92a48681f909e113ad7ddb253817ccc87de358db6cd37cd63 not found: ID does not exist" Oct 30 06:22:48.027536 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.027441 5565 scope.go:122] "RemoveContainer" containerID="930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692" Oct 30 06:22:48.037016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.037001 5565 scope.go:122] "RemoveContainer" containerID="930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692" Oct 30 06:22:48.037255 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:22:48.037235 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692\": container with ID starting with 930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692 not found: ID does not exist" containerID="930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692" Oct 30 06:22:48.037315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.037261 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692"} err="failed to get container status \"930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692\": rpc error: code = NotFound desc = could not find container \"930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692\": container with ID starting with 930da70a003f4b342da8f14f178a2fdbe4741b3bc91bd5d0d25bb98350a17692 not found: ID does not exist" Oct 30 06:22:48.072944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.072909 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2"] Oct 30 06:22:48.077085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.077055 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-7fdb95-5f78b9d54d-vdtc2"] Oct 30 06:22:48.114509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.114472 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962"] Oct 30 06:22:48.123176 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.123145 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-7fdb95-7b5565664c-9n962"] Oct 30 06:22:48.445114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.445015 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd"] Oct 30 06:22:48.447748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:48.447726 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-a14091-vzrzd"] Oct 30 06:22:49.570093 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:49.570059 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="649f0a68-7af7-49e6-a6d0-39afce89434a" path="/var/lib/kubelet/pods/649f0a68-7af7-49e6-a6d0-39afce89434a/volumes" Oct 30 06:22:49.570760 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:49.570637 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9235d4bf-58b3-4a2b-8e86-b915affdb1d8" path="/var/lib/kubelet/pods/9235d4bf-58b3-4a2b-8e86-b915affdb1d8/volumes" Oct 30 06:22:49.571024 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:49.571006 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b159bb4f-a575-4cae-8dc4-e6f5582fc629" path="/var/lib/kubelet/pods/b159bb4f-a575-4cae-8dc4-e6f5582fc629/volumes" Oct 30 06:22:49.571374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:49.571359 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca564ba7-731b-4d1d-b155-1dd05c326809" path="/var/lib/kubelet/pods/ca564ba7-731b-4d1d-b155-1dd05c326809/volumes" Oct 30 06:22:50.438084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:50.438041 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj"] Oct 30 06:22:51.224642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:51.224606 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" podUID="820c0009-c7e3-4035-a4ba-733b27571362" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:51.225104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:51.224606 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:22:55.438716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:55.438678 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerName="payload-pre-processing" containerID="cri-o://092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4" gracePeriod=25 Oct 30 06:22:56.081547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:56.081209 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s"] Oct 30 06:22:56.081547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:56.081451 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" containerID="cri-o://f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8" gracePeriod=30 Oct 30 06:22:58.684078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:58.684043 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68"] Oct 30 06:22:58.684573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:58.684256 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" containerID="cri-o://a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4" gracePeriod=30 Oct 30 06:22:59.489545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:59.488566 5565 patch_prober.go:29] interesting pod/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:22:59.489545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:59.488635 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:22:59.516248 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:59.516213 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn"] Oct 30 06:22:59.516582 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:22:59.516520 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" containerID="cri-o://434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759" gracePeriod=30 Oct 30 06:23:01.224507 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.224467 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" podUID="820c0009-c7e3-4035-a4ba-733b27571362" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:01.224995 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.224973 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:01.285743 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.285710 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7"] Oct 30 06:23:01.285990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.285961 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" podUID="14da1d18-aa12-4d44-8a55-5217833d0686" containerName="maas-api" containerID="cri-o://a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106" gracePeriod=30 Oct 30 06:23:01.423373 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.423347 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 06:23:01.525426 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525388 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.525609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525553 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.525609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525592 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.525779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525624 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.525779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525685 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.525779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525757 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.526014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525811 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.526014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525850 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.526014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525897 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket\") pod \"f5490d6a-3257-42ba-83cf-dc71061f6835\" (UID: \"f5490d6a-3257-42ba-83cf-dc71061f6835\") " Oct 30 06:23:01.526014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.525930 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.526304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.526179 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.526431 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.526410 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.526564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.526503 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.526564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.526550 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.526803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.526734 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:01.527171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.527146 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.530009 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.529981 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "kube-api-access-vddrh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:01.530313 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.530288 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:01.530464 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.530432 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:01.532064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.532041 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy" pod "f5490d6a-3257-42ba-83cf-dc71061f6835" (UID: "f5490d6a-3257-42ba-83cf-dc71061f6835"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:01.568834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.568804 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:23:01.628011 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.627938 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs\") pod \"14da1d18-aa12-4d44-8a55-5217833d0686\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " Oct 30 06:23:01.628154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.628130 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca\") pod \"14da1d18-aa12-4d44-8a55-5217833d0686\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " Oct 30 06:23:01.628226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.628212 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls\") pod \"14da1d18-aa12-4d44-8a55-5217833d0686\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " Oct 30 06:23:01.628357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.628334 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg\" (UniqueName: \"kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg\") pod \"14da1d18-aa12-4d44-8a55-5217833d0686\" (UID: \"14da1d18-aa12-4d44-8a55-5217833d0686\") " Oct 30 06:23:01.628483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.628456 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca" pod "14da1d18-aa12-4d44-8a55-5217833d0686" (UID: "14da1d18-aa12-4d44-8a55-5217833d0686"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.628985 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/14da1d18-aa12-4d44-8a55-5217833d0686-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629013 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f5490d6a-3257-42ba-83cf-dc71061f6835-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629028 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vddrh\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-kube-api-access-vddrh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629043 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629058 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629070 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629083 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.629129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.629097 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f5490d6a-3257-42ba-83cf-dc71061f6835-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.631056 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.630998 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls" pod "14da1d18-aa12-4d44-8a55-5217833d0686" (UID: "14da1d18-aa12-4d44-8a55-5217833d0686"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:01.631227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.631073 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg" pod "14da1d18-aa12-4d44-8a55-5217833d0686" (UID: "14da1d18-aa12-4d44-8a55-5217833d0686"). InnerVolumeSpecName "kube-api-access-mdkjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:01.631227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.631129 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs" pod "14da1d18-aa12-4d44-8a55-5217833d0686" (UID: "14da1d18-aa12-4d44-8a55-5217833d0686"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:01.634636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.634612 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4"] Oct 30 06:23:01.634805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.634785 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" podUID="cbcd69ad-b321-40f0-a2b8-691dcaab5011" containerName="payload-processing" containerID="cri-o://fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6" gracePeriod=30 Oct 30 06:23:01.655114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.654086 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd"] Oct 30 06:23:01.655480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.655448 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" podUID="4a03a379-915f-49d6-98e5-a6aaa7275e7b" containerName="payload-pre-processing" containerID="cri-o://59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9" gracePeriod=30 Oct 30 06:23:01.732860 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.732826 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdkjg\" (UniqueName: \"kubernetes.io/projected/14da1d18-aa12-4d44-8a55-5217833d0686-kube-api-access-mdkjg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.732983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.732865 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.732983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.732880 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/14da1d18-aa12-4d44-8a55-5217833d0686-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:01.948277 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:01.947983 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:23:02.007496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.007464 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:23:02.047813 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.047784 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume\") pod \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " Oct 30 06:23:02.047983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.047835 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r\" (UniqueName: \"kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r\") pod \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\" (UID: \"cbcd69ad-b321-40f0-a2b8-691dcaab5011\") " Oct 30 06:23:02.048156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.048134 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume" pod "cbcd69ad-b321-40f0-a2b8-691dcaab5011" (UID: "cbcd69ad-b321-40f0-a2b8-691dcaab5011"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:02.048330 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.048308 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbcd69ad-b321-40f0-a2b8-691dcaab5011-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.049916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.049896 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r" pod "cbcd69ad-b321-40f0-a2b8-691dcaab5011" (UID: "cbcd69ad-b321-40f0-a2b8-691dcaab5011"). InnerVolumeSpecName "kube-api-access-jdn9r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:02.136715 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.136684 5565 generic.go:356] "Generic (PLEG): container finished" podID="4a03a379-915f-49d6-98e5-a6aaa7275e7b" containerID="59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9" exitCode=0 Oct 30 06:23:02.136905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.136735 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" Oct 30 06:23:02.136905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.136788 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" event={"ID":"4a03a379-915f-49d6-98e5-a6aaa7275e7b","Type":"ContainerDied","Data":"59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9"} Oct 30 06:23:02.136905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.136859 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd" event={"ID":"4a03a379-915f-49d6-98e5-a6aaa7275e7b","Type":"ContainerDied","Data":"958bd4ffe8d3ffe1195d64c1d066101cb3430ad809b399bd17ef6f60342eb2d4"} Oct 30 06:23:02.137095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.136929 5565 scope.go:122] "RemoveContainer" containerID="59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9" Oct 30 06:23:02.138597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.138466 5565 generic.go:356] "Generic (PLEG): container finished" podID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerID="f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8" exitCode=0 Oct 30 06:23:02.138597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.138559 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" Oct 30 06:23:02.138782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.138597 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" event={"ID":"f5490d6a-3257-42ba-83cf-dc71061f6835","Type":"ContainerDied","Data":"f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8"} Oct 30 06:23:02.138782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.138665 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s" event={"ID":"f5490d6a-3257-42ba-83cf-dc71061f6835","Type":"ContainerDied","Data":"1e747f10b2f173e801f9452f2ed77b8d9a6aac5aca43ec82062cd15faa6dfaa4"} Oct 30 06:23:02.140472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.140438 5565 generic.go:356] "Generic (PLEG): container finished" podID="14da1d18-aa12-4d44-8a55-5217833d0686" containerID="a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106" exitCode=0 Oct 30 06:23:02.140627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.140475 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" event={"ID":"14da1d18-aa12-4d44-8a55-5217833d0686","Type":"ContainerDied","Data":"a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106"} Oct 30 06:23:02.140627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.140517 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" event={"ID":"14da1d18-aa12-4d44-8a55-5217833d0686","Type":"ContainerDied","Data":"6fffeaaa7f4245bf5fdcd48bfc7c9e43e838d9cb41833b8b03cde0637534a066"} Oct 30 06:23:02.140627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.140486 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7" Oct 30 06:23:02.142283 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.142255 5565 generic.go:356] "Generic (PLEG): container finished" podID="cbcd69ad-b321-40f0-a2b8-691dcaab5011" containerID="fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6" exitCode=0 Oct 30 06:23:02.142422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.142293 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" Oct 30 06:23:02.142422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.142338 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" event={"ID":"cbcd69ad-b321-40f0-a2b8-691dcaab5011","Type":"ContainerDied","Data":"fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6"} Oct 30 06:23:02.142422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.142392 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4" event={"ID":"cbcd69ad-b321-40f0-a2b8-691dcaab5011","Type":"ContainerDied","Data":"c0fa4816008718cd48b972c1da578d20a5ccf65e836839197ee3e166398bbc42"} Oct 30 06:23:02.149075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.149049 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn\" (UniqueName: \"kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn\") pod \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " Oct 30 06:23:02.149558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.149281 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume\") pod \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\" (UID: \"4a03a379-915f-49d6-98e5-a6aaa7275e7b\") " Oct 30 06:23:02.149919 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.149897 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdn9r\" (UniqueName: \"kubernetes.io/projected/cbcd69ad-b321-40f0-a2b8-691dcaab5011-kube-api-access-jdn9r\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.150026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.150003 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume" pod "4a03a379-915f-49d6-98e5-a6aaa7275e7b" (UID: "4a03a379-915f-49d6-98e5-a6aaa7275e7b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:02.152267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.152249 5565 scope.go:122] "RemoveContainer" containerID="59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9" Oct 30 06:23:02.152393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.152279 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn" pod "4a03a379-915f-49d6-98e5-a6aaa7275e7b" (UID: "4a03a379-915f-49d6-98e5-a6aaa7275e7b"). InnerVolumeSpecName "kube-api-access-hqwjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:02.152572 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:02.152551 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9\": container with ID starting with 59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9 not found: ID does not exist" containerID="59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9" Oct 30 06:23:02.152637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.152579 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9"} err="failed to get container status \"59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9\": rpc error: code = NotFound desc = could not find container \"59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9\": container with ID starting with 59ff885cbce3b810273fab0890951c284e82f7d7646d607a9d3683bc47b74da9 not found: ID does not exist" Oct 30 06:23:02.152637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.152598 5565 scope.go:122] "RemoveContainer" containerID="f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8" Oct 30 06:23:02.171179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.168400 5565 scope.go:122] "RemoveContainer" containerID="f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8" Oct 30 06:23:02.171179 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:02.168775 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8\": container with ID starting with f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8 not found: ID does not exist" containerID="f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8" Oct 30 06:23:02.171179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.168808 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8"} err="failed to get container status \"f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8\": rpc error: code = NotFound desc = could not find container \"f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8\": container with ID starting with f2f14093410b526a6a22b0f74c0c4dcebad44277681e18986fe8f5a7daebbdd8 not found: ID does not exist" Oct 30 06:23:02.171179 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.168831 5565 scope.go:122] "RemoveContainer" containerID="a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106" Oct 30 06:23:02.178583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.178566 5565 scope.go:122] "RemoveContainer" containerID="a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106" Oct 30 06:23:02.178862 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:02.178843 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106\": container with ID starting with a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106 not found: ID does not exist" containerID="a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106" Oct 30 06:23:02.178962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.178873 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106"} err="failed to get container status \"a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106\": rpc error: code = NotFound desc = could not find container \"a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106\": container with ID starting with a80184769578fbb50da8d6abff05e69df4480b9697312c0b06be54586edc0106 not found: ID does not exist" Oct 30 06:23:02.178962 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.178895 5565 scope.go:122] "RemoveContainer" containerID="fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6" Oct 30 06:23:02.188429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.188398 5565 scope.go:122] "RemoveContainer" containerID="fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6" Oct 30 06:23:02.188732 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:02.188714 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6\": container with ID starting with fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6 not found: ID does not exist" containerID="fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6" Oct 30 06:23:02.188783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.188740 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6"} err="failed to get container status \"fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6\": rpc error: code = NotFound desc = could not find container \"fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6\": container with ID starting with fbd14a6f89c9d82720063d0463dcf8ce700931a9b3a497bb9f51d2d8d53492a6 not found: ID does not exist" Oct 30 06:23:02.230894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.230870 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s"] Oct 30 06:23:02.235990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.235965 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-a14091-openshift-default-58fbd668d5-9cd9s"] Oct 30 06:23:02.241063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.241041 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl"] Oct 30 06:23:02.247662 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.247640 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d4bace-vq4kl"] Oct 30 06:23:02.250919 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.250895 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqwjn\" (UniqueName: \"kubernetes.io/projected/4a03a379-915f-49d6-98e5-a6aaa7275e7b-kube-api-access-hqwjn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.251018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.250925 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4a03a379-915f-49d6-98e5-a6aaa7275e7b-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:02.264619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.264593 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:02.273731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.273704 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7"] Oct 30 06:23:02.279885 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.279863 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-d4bace-796d864687-ffdp7"] Oct 30 06:23:02.317836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.317801 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4"] Oct 30 06:23:02.324462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.324430 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d4bace-689cbd644b-hk5r4"] Oct 30 06:23:02.487151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.487121 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd"] Oct 30 06:23:02.491708 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:02.491682 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d4bace-57b888f76c-wtrjd"] Oct 30 06:23:03.567389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.567357 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14da1d18-aa12-4d44-8a55-5217833d0686" path="/var/lib/kubelet/pods/14da1d18-aa12-4d44-8a55-5217833d0686/volumes" Oct 30 06:23:03.567796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.567772 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a03a379-915f-49d6-98e5-a6aaa7275e7b" path="/var/lib/kubelet/pods/4a03a379-915f-49d6-98e5-a6aaa7275e7b/volumes" Oct 30 06:23:03.568131 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.568117 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbcd69ad-b321-40f0-a2b8-691dcaab5011" path="/var/lib/kubelet/pods/cbcd69ad-b321-40f0-a2b8-691dcaab5011/volumes" Oct 30 06:23:03.568628 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.568608 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf861b9e-af63-4f3e-9858-71c398e67cdd" path="/var/lib/kubelet/pods/cf861b9e-af63-4f3e-9858-71c398e67cdd/volumes" Oct 30 06:23:03.569143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.569127 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" path="/var/lib/kubelet/pods/f5490d6a-3257-42ba-83cf-dc71061f6835/volumes" Oct 30 06:23:03.801085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.801058 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:23:03.953200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:03.953177 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:23:04.071099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071060 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071107 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071150 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071337 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071403 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071451 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.071565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071461 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.071565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071485 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071565 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071539 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071651 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071655 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.071802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071740 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert\") pod \"d6437fe8-d09c-419d-ac5b-33f689da1554\" (UID: \"d6437fe8-d09c-419d-ac5b-33f689da1554\") " Oct 30 06:23:04.071958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.071812 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.072169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.072132 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:04.072313 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.072293 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6437fe8-d09c-419d-ac5b-33f689da1554-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.072402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.072327 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.072402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.072342 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.072402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.072358 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.072402 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.072374 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.073831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.073805 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:04.074117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.074096 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.074175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.074119 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:04.074240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.074222 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9" pod "d6437fe8-d09c-419d-ac5b-33f689da1554" (UID: "d6437fe8-d09c-419d-ac5b-33f689da1554"). InnerVolumeSpecName "kube-api-access-97cp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:04.170136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.167586 5565 generic.go:356] "Generic (PLEG): container finished" podID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerID="a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4" exitCode=0 Oct 30 06:23:04.170136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.167724 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" event={"ID":"d6437fe8-d09c-419d-ac5b-33f689da1554","Type":"ContainerDied","Data":"a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4"} Oct 30 06:23:04.170136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.167778 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" event={"ID":"d6437fe8-d09c-419d-ac5b-33f689da1554","Type":"ContainerDied","Data":"89f797408788950eac7910f2d31d8a9f768b5c0dced080366386b493668333eb"} Oct 30 06:23:04.170136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.167808 5565 scope.go:122] "RemoveContainer" containerID="a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4" Oct 30 06:23:04.170136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.167994 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68" Oct 30 06:23:04.173773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.173732 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.173773 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.173774 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.174018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.173791 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97cp9\" (UniqueName: \"kubernetes.io/projected/d6437fe8-d09c-419d-ac5b-33f689da1554-kube-api-access-97cp9\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.174018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.173811 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6437fe8-d09c-419d-ac5b-33f689da1554-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.184704 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.184681 5565 scope.go:122] "RemoveContainer" containerID="a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4" Oct 30 06:23:04.185164 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:04.185141 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4\": container with ID starting with a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4 not found: ID does not exist" containerID="a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4" Oct 30 06:23:04.185264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.185175 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4"} err="failed to get container status \"a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4\": rpc error: code = NotFound desc = could not find container \"a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4\": container with ID starting with a7c2f5e3dc7490d413da94392641c9b6a122c7863c5978d6cd29e5f65085cfb4 not found: ID does not exist" Oct 30 06:23:04.229897 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.229801 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68"] Oct 30 06:23:04.235182 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.235151 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-95498c-openshift-default-5f44545945-krg68"] Oct 30 06:23:04.494468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.494433 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:23:04.551326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.551287 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:23:04.579560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.579501 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume\") pod \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " Oct 30 06:23:04.580214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.579948 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume" pod "2795ff00-a5e2-4bb0-baa7-70a86a50429d" (UID: "2795ff00-a5e2-4bb0-baa7-70a86a50429d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:04.580357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.580265 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8\" (UniqueName: \"kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8\") pod \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\" (UID: \"2795ff00-a5e2-4bb0-baa7-70a86a50429d\") " Oct 30 06:23:04.581010 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.580981 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2795ff00-a5e2-4bb0-baa7-70a86a50429d-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.582840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.582805 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8" pod "2795ff00-a5e2-4bb0-baa7-70a86a50429d" (UID: "2795ff00-a5e2-4bb0-baa7-70a86a50429d"). InnerVolumeSpecName "kube-api-access-642v8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:04.682174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.682104 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7\" (UniqueName: \"kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7\") pod \"820c0009-c7e3-4035-a4ba-733b27571362\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " Oct 30 06:23:04.682624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.682556 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume\") pod \"820c0009-c7e3-4035-a4ba-733b27571362\" (UID: \"820c0009-c7e3-4035-a4ba-733b27571362\") " Oct 30 06:23:04.683479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.683382 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-642v8\" (UniqueName: \"kubernetes.io/projected/2795ff00-a5e2-4bb0-baa7-70a86a50429d-kube-api-access-642v8\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.684353 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.684275 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume" pod "820c0009-c7e3-4035-a4ba-733b27571362" (UID: "820c0009-c7e3-4035-a4ba-733b27571362"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:04.701101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.701006 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7" pod "820c0009-c7e3-4035-a4ba-733b27571362" (UID: "820c0009-c7e3-4035-a4ba-733b27571362"). InnerVolumeSpecName "kube-api-access-95qz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:04.784844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.784799 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/820c0009-c7e3-4035-a4ba-733b27571362-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.784844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.784844 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95qz7\" (UniqueName: \"kubernetes.io/projected/820c0009-c7e3-4035-a4ba-733b27571362-kube-api-access-95qz7\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.868912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.868882 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:23:04.976344 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.976242 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz"] Oct 30 06:23:04.986608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986411 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986481 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986584 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986612 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986639 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986692 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986717 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986748 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.986989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.986812 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k\") pod \"61891b18-0cc3-42d4-9866-8da70e8e06e1\" (UID: \"61891b18-0cc3-42d4-9866-8da70e8e06e1\") " Oct 30 06:23:04.987291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.987263 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.987440 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.987422 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:04.987844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.987822 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.989650 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.989620 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:04.989774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.989695 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.989837 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.989824 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "kube-api-access-8br5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:04.990655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.990627 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.991004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.990983 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:04.991305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.991276 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p"] Oct 30 06:23:04.991865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.991843 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:04.993477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:04.993455 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo" pod "61891b18-0cc3-42d4-9866-8da70e8e06e1" (UID: "61891b18-0cc3-42d4-9866-8da70e8e06e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:05.088200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088158 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088194 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088204 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088212 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088220 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61891b18-0cc3-42d4-9866-8da70e8e06e1-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088228 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61891b18-0cc3-42d4-9866-8da70e8e06e1-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088239 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61891b18-0cc3-42d4-9866-8da70e8e06e1-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.088513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.088250 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8br5k\" (UniqueName: \"kubernetes.io/projected/61891b18-0cc3-42d4-9866-8da70e8e06e1-kube-api-access-8br5k\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:05.175459 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.175423 5565 generic.go:356] "Generic (PLEG): container finished" podID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerID="434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759" exitCode=0 Oct 30 06:23:05.175649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.175473 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" Oct 30 06:23:05.175649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.175505 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" event={"ID":"61891b18-0cc3-42d4-9866-8da70e8e06e1","Type":"ContainerDied","Data":"434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759"} Oct 30 06:23:05.175649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.175604 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn" event={"ID":"61891b18-0cc3-42d4-9866-8da70e8e06e1","Type":"ContainerDied","Data":"947201b8631729064c5198cf547b4a221dcb5d3e4c01ee5ebbf30b8b5d25174d"} Oct 30 06:23:05.175649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.175638 5565 scope.go:122] "RemoveContainer" containerID="434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759" Oct 30 06:23:05.177140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.177121 5565 generic.go:356] "Generic (PLEG): container finished" podID="820c0009-c7e3-4035-a4ba-733b27571362" containerID="93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e" exitCode=0 Oct 30 06:23:05.177233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.177168 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" Oct 30 06:23:05.177233 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.177210 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" event={"ID":"820c0009-c7e3-4035-a4ba-733b27571362","Type":"ContainerDied","Data":"93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e"} Oct 30 06:23:05.177331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.177240 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd" event={"ID":"820c0009-c7e3-4035-a4ba-733b27571362","Type":"ContainerDied","Data":"fccff1111c8fb5e3ffcc1b77e6f6c696899b12c4d3db9eddac3d26869e1fa672"} Oct 30 06:23:05.179835 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.179810 5565 generic.go:356] "Generic (PLEG): container finished" podID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerID="5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b" exitCode=0 Oct 30 06:23:05.179930 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.179861 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" Oct 30 06:23:05.179975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.179864 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" event={"ID":"2795ff00-a5e2-4bb0-baa7-70a86a50429d","Type":"ContainerDied","Data":"5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b"} Oct 30 06:23:05.180026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.179979 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5" event={"ID":"2795ff00-a5e2-4bb0-baa7-70a86a50429d","Type":"ContainerDied","Data":"71eceb1b156069057b17ef4fda48289598e19a4526f0980f25d7e613a0ed007e"} Oct 30 06:23:05.189769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.189721 5565 scope.go:122] "RemoveContainer" containerID="434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759" Oct 30 06:23:05.190115 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:05.190091 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759\": container with ID starting with 434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759 not found: ID does not exist" containerID="434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759" Oct 30 06:23:05.190211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.190124 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759"} err="failed to get container status \"434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759\": rpc error: code = NotFound desc = could not find container \"434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759\": container with ID starting with 434abe513d41869996e229c5924740fc45d54fc24ccf8105a93dae3ced7a7759 not found: ID does not exist" Oct 30 06:23:05.190211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.190150 5565 scope.go:122] "RemoveContainer" containerID="93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e" Oct 30 06:23:05.201234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.201215 5565 scope.go:122] "RemoveContainer" containerID="93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e" Oct 30 06:23:05.201555 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:05.201519 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e\": container with ID starting with 93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e not found: ID does not exist" containerID="93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e" Oct 30 06:23:05.201620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.201564 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e"} err="failed to get container status \"93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e\": rpc error: code = NotFound desc = could not find container \"93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e\": container with ID starting with 93efe6f2da5a6b13b190a758675ac8da1a52057c8c166865ab0c444ffa8fe73e not found: ID does not exist" Oct 30 06:23:05.201620 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.201586 5565 scope.go:122] "RemoveContainer" containerID="5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b" Oct 30 06:23:05.211495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.211472 5565 scope.go:122] "RemoveContainer" containerID="5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b" Oct 30 06:23:05.211875 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:05.211854 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b\": container with ID starting with 5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b not found: ID does not exist" containerID="5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b" Oct 30 06:23:05.211941 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.211885 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b"} err="failed to get container status \"5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b\": rpc error: code = NotFound desc = could not find container \"5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b\": container with ID starting with 5609d9293d0f4a1a374cbf316fca2ba2be9dcbbef020fb2dada65aad7402681b not found: ID does not exist" Oct 30 06:23:05.232562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.232462 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5"] Oct 30 06:23:05.238685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.238656 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-6b8f87-7b9d69466-84nq5"] Oct 30 06:23:05.278828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.278785 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn"] Oct 30 06:23:05.283272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.283240 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-6b8f87-openshift-default-5b68b78d7c-5svbn"] Oct 30 06:23:05.320832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.320788 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd"] Oct 30 06:23:05.325510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.325459 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-6b8f87-db4c876f9-cjnjd"] Oct 30 06:23:05.568370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.568332 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" path="/var/lib/kubelet/pods/2795ff00-a5e2-4bb0-baa7-70a86a50429d/volumes" Oct 30 06:23:05.569034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.569008 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" path="/var/lib/kubelet/pods/61891b18-0cc3-42d4-9866-8da70e8e06e1/volumes" Oct 30 06:23:05.569421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.569403 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="820c0009-c7e3-4035-a4ba-733b27571362" path="/var/lib/kubelet/pods/820c0009-c7e3-4035-a4ba-733b27571362/volumes" Oct 30 06:23:05.570211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:05.570195 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" path="/var/lib/kubelet/pods/d6437fe8-d09c-419d-ac5b-33f689da1554/volumes" Oct 30 06:23:09.628599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.628566 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj"] Oct 30 06:23:09.629062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629048 5565 cpu_manager.go:414] "containerMap: removing container" podUID="820c0009-c7e3-4035-a4ba-733b27571362" containerName="payload-pre-processing" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629062 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="820c0009-c7e3-4035-a4ba-733b27571362" containerName="payload-pre-processing" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629087 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ef280ae8-0bc8-458a-bf67-b9a09ba41aec" containerName="revoke-keys" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629092 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef280ae8-0bc8-458a-bf67-b9a09ba41aec" containerName="revoke-keys" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629103 5565 cpu_manager.go:414] "containerMap: removing container" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629108 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="121ba31d-7c52-4d21-bac3-fc3848d2a8f4" containerName="istio-proxy" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629116 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cbcd69ad-b321-40f0-a2b8-691dcaab5011" containerName="payload-processing" Oct 30 06:23:09.629121 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629120 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbcd69ad-b321-40f0-a2b8-691dcaab5011" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629127 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4a03a379-915f-49d6-98e5-a6aaa7275e7b" containerName="payload-pre-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629132 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a03a379-915f-49d6-98e5-a6aaa7275e7b" containerName="payload-pre-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629141 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b159bb4f-a575-4cae-8dc4-e6f5582fc629" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629146 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b159bb4f-a575-4cae-8dc4-e6f5582fc629" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629153 5565 cpu_manager.go:414] "containerMap: removing container" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629157 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629164 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629168 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629185 5565 cpu_manager.go:414] "containerMap: removing container" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629190 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629199 5565 cpu_manager.go:414] "containerMap: removing container" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629204 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629219 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ca564ba7-731b-4d1d-b155-1dd05c326809" containerName="payload-pre-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629224 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca564ba7-731b-4d1d-b155-1dd05c326809" containerName="payload-pre-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629232 5565 cpu_manager.go:414] "containerMap: removing container" podUID="14da1d18-aa12-4d44-8a55-5217833d0686" containerName="maas-api" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629236 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14da1d18-aa12-4d44-8a55-5217833d0686" containerName="maas-api" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629309 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef280ae8-0bc8-458a-bf67-b9a09ba41aec" containerName="revoke-keys" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629320 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbcd69ad-b321-40f0-a2b8-691dcaab5011" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629326 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a03a379-915f-49d6-98e5-a6aaa7275e7b" containerName="payload-pre-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629331 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b159bb4f-a575-4cae-8dc4-e6f5582fc629" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629337 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="2795ff00-a5e2-4bb0-baa7-70a86a50429d" containerName="payload-processing" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629341 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6437fe8-d09c-419d-ac5b-33f689da1554" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629349 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="61891b18-0cc3-42d4-9866-8da70e8e06e1" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629355 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5490d6a-3257-42ba-83cf-dc71061f6835" containerName="istio-proxy" Oct 30 06:23:09.629350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629361 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5b7b1b7-e95c-46b9-b61a-86fb98a3c6ea" containerName="istio-proxy" Oct 30 06:23:09.630171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629368 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca564ba7-731b-4d1d-b155-1dd05c326809" containerName="payload-pre-processing" Oct 30 06:23:09.630171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629374 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="14da1d18-aa12-4d44-8a55-5217833d0686" containerName="maas-api" Oct 30 06:23:09.630171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.629381 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="820c0009-c7e3-4035-a4ba-733b27571362" containerName="payload-pre-processing" Oct 30 06:23:09.633657 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.633628 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.650586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.650558 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.650851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.650813 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.651122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.651094 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s7hg\" (UniqueName: \"kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.653353 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.653329 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj"] Oct 30 06:23:09.702230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.702198 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:09.702230 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.702202 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:23:09.752772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.752734 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s7hg\" (UniqueName: \"kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.752929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.752795 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.752929 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.752830 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.753426 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.753402 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.755158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.755134 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.777755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.777726 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s7hg\" (UniqueName: \"kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:09.946863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:09.946780 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:10.247292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:10.247264 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj"] Oct 30 06:23:10.249623 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:23:10.249586 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54406b29_9815_42e7_afdb_78ca593fd477.slice/crio-2567255e0a722e13c7283844ba28c40db1780b47a1c990a65381b9a054d03a77 WatchSource:0}: Error finding container 2567255e0a722e13c7283844ba28c40db1780b47a1c990a65381b9a054d03a77: Status 404 returned error can't find the container with id 2567255e0a722e13c7283844ba28c40db1780b47a1c990a65381b9a054d03a77 Oct 30 06:23:11.262635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:11.262605 5565 generic.go:356] "Generic (PLEG): container finished" podID="54406b29-9815-42e7-afdb-78ca593fd477" containerID="fced9fd236e7bd4a9488353be02aecfc65575038225ec0a33670610173e9ff06" exitCode=0 Oct 30 06:23:11.263081 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:11.262687 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" event={"ID":"54406b29-9815-42e7-afdb-78ca593fd477","Type":"ContainerDied","Data":"fced9fd236e7bd4a9488353be02aecfc65575038225ec0a33670610173e9ff06"} Oct 30 06:23:11.263081 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:11.262732 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" event={"ID":"54406b29-9815-42e7-afdb-78ca593fd477","Type":"ContainerStarted","Data":"2567255e0a722e13c7283844ba28c40db1780b47a1c990a65381b9a054d03a77"} Oct 30 06:23:12.265232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.265196 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:23:12.408329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.408293 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:12.489548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.488996 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca\") pod \"54406b29-9815-42e7-afdb-78ca593fd477\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " Oct 30 06:23:12.489548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.489079 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca\") pod \"54406b29-9815-42e7-afdb-78ca593fd477\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " Oct 30 06:23:12.489548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.489108 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg\" (UniqueName: \"kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg\") pod \"54406b29-9815-42e7-afdb-78ca593fd477\" (UID: \"54406b29-9815-42e7-afdb-78ca593fd477\") " Oct 30 06:23:12.489548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.489463 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca" pod "54406b29-9815-42e7-afdb-78ca593fd477" (UID: "54406b29-9815-42e7-afdb-78ca593fd477"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:12.494122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.492893 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg" pod "54406b29-9815-42e7-afdb-78ca593fd477" (UID: "54406b29-9815-42e7-afdb-78ca593fd477"). InnerVolumeSpecName "kube-api-access-9s7hg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:12.494806 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.494782 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca" pod "54406b29-9815-42e7-afdb-78ca593fd477" (UID: "54406b29-9815-42e7-afdb-78ca593fd477"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:12.592540 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.592430 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.592540 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.592474 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/54406b29-9815-42e7-afdb-78ca593fd477-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.592540 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.592488 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s7hg\" (UniqueName: \"kubernetes.io/projected/54406b29-9815-42e7-afdb-78ca593fd477-kube-api-access-9s7hg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:12.924014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.923928 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7"] Oct 30 06:23:12.927272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:12.927241 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-3abe42-zmcb7"] Oct 30 06:23:13.277056 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:13.277032 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" Oct 30 06:23:13.277430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:13.277065 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj" event={"ID":"54406b29-9815-42e7-afdb-78ca593fd477","Type":"ContainerDied","Data":"2567255e0a722e13c7283844ba28c40db1780b47a1c990a65381b9a054d03a77"} Oct 30 06:23:13.277430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:13.277117 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2567255e0a722e13c7283844ba28c40db1780b47a1c990a65381b9a054d03a77" Oct 30 06:23:13.567513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:13.567437 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef280ae8-0bc8-458a-bf67-b9a09ba41aec" path="/var/lib/kubelet/pods/ef280ae8-0bc8-458a-bf67-b9a09ba41aec/volumes" Oct 30 06:23:15.711325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.711268 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 06:23:15.824923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.824893 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume\") pod \"5d17330c-7e57-4aa1-b3b2-59301813307f\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " Oct 30 06:23:15.825083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.825022 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc\" (UniqueName: \"kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc\") pod \"5d17330c-7e57-4aa1-b3b2-59301813307f\" (UID: \"5d17330c-7e57-4aa1-b3b2-59301813307f\") " Oct 30 06:23:15.825315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.825293 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume" pod "5d17330c-7e57-4aa1-b3b2-59301813307f" (UID: "5d17330c-7e57-4aa1-b3b2-59301813307f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:15.825470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.825457 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d17330c-7e57-4aa1-b3b2-59301813307f-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:15.827078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.827048 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc" pod "5d17330c-7e57-4aa1-b3b2-59301813307f" (UID: "5d17330c-7e57-4aa1-b3b2-59301813307f"). InnerVolumeSpecName "kube-api-access-d2xwc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:15.927091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:15.927064 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2xwc\" (UniqueName: \"kubernetes.io/projected/5d17330c-7e57-4aa1-b3b2-59301813307f-kube-api-access-d2xwc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:16.297236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.297202 5565 generic.go:356] "Generic (PLEG): container finished" podID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerID="092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4" exitCode=0 Oct 30 06:23:16.297446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.297248 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" Oct 30 06:23:16.297446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.297289 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" event={"ID":"5d17330c-7e57-4aa1-b3b2-59301813307f","Type":"ContainerDied","Data":"092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4"} Oct 30 06:23:16.297446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.297344 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj" event={"ID":"5d17330c-7e57-4aa1-b3b2-59301813307f","Type":"ContainerDied","Data":"a716b7b0b6fd13f560daf478ac07d032c114237a79976933a76f05e1a77c9c8b"} Oct 30 06:23:16.297446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.297374 5565 scope.go:122] "RemoveContainer" containerID="092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4" Oct 30 06:23:16.308224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.308207 5565 scope.go:122] "RemoveContainer" containerID="092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4" Oct 30 06:23:16.308508 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:16.308487 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4\": container with ID starting with 092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4 not found: ID does not exist" containerID="092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4" Oct 30 06:23:16.308584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.308515 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4"} err="failed to get container status \"092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4\": rpc error: code = NotFound desc = could not find container \"092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4\": container with ID starting with 092bd8b36790fb4bb8486bf9318422fd307cdf5fa458624ed24be1dc271ddae4 not found: ID does not exist" Oct 30 06:23:16.354496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.354474 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj"] Oct 30 06:23:16.368672 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:16.368649 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-3abe42-9f5d77579-cjwzj"] Oct 30 06:23:17.531765 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:17.531726 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:23:17.567350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:17.567311 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" path="/var/lib/kubelet/pods/5d17330c-7e57-4aa1-b3b2-59301813307f/volumes" Oct 30 06:23:20.001654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.001611 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" containerID="cri-o://12f1d15106a4c3ca7665857e14f5894e7384012b31f6234d27b886798118ade1" gracePeriod=45 Oct 30 06:23:20.013981 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.013942 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" containerID="cri-o://9e17c784f73ada8f933c48ce9e50b7a8c9bd8761d701982483f74419d8afae98" gracePeriod=45 Oct 30 06:23:20.199757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.199722 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx"] Oct 30 06:23:20.199973 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.199947 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" containerID="cri-o://7dc99a69d1a41e71c2c0959b848c7f06e5e9201e4ae2e735671aaadb3110d6d6" gracePeriod=30 Oct 30 06:23:20.365654 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.365621 5565 generic.go:356] "Generic (PLEG): container finished" podID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerID="9e17c784f73ada8f933c48ce9e50b7a8c9bd8761d701982483f74419d8afae98" exitCode=0 Oct 30 06:23:20.365872 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.365726 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" event={"ID":"be07924b-4a72-4f8f-8ff0-e4e71854b388","Type":"ContainerDied","Data":"9e17c784f73ada8f933c48ce9e50b7a8c9bd8761d701982483f74419d8afae98"} Oct 30 06:23:20.367595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.367562 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" event={"ID":"29a9f246-2ac1-4eb4-a230-8d028068b3dd","Type":"ContainerDied","Data":"12f1d15106a4c3ca7665857e14f5894e7384012b31f6234d27b886798118ade1"} Oct 30 06:23:20.367727 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.367566 5565 generic.go:356] "Generic (PLEG): container finished" podID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerID="12f1d15106a4c3ca7665857e14f5894e7384012b31f6234d27b886798118ade1" exitCode=0 Oct 30 06:23:20.384598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.384575 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 06:23:20.388397 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.388375 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 06:23:20.485996 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.485956 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn\" (UniqueName: \"kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn\") pod \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " Oct 30 06:23:20.486188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486034 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs\" (UniqueName: \"kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs\") pod \"be07924b-4a72-4f8f-8ff0-e4e71854b388\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " Oct 30 06:23:20.486188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486073 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm\") pod \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " Oct 30 06:23:20.486188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486139 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df\" (UniqueName: \"kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df\") pod \"be07924b-4a72-4f8f-8ff0-e4e71854b388\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " Oct 30 06:23:20.486188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486169 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache\") pod \"be07924b-4a72-4f8f-8ff0-e4e71854b388\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " Oct 30 06:23:20.486418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486199 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs\" (UniqueName: \"kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs\") pod \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " Oct 30 06:23:20.486418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486261 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir\") pod \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " Oct 30 06:23:20.486418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486319 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache\") pod \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " Oct 30 06:23:20.486418 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486406 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home\") pod \"be07924b-4a72-4f8f-8ff0-e4e71854b388\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " Oct 30 06:23:20.486668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486432 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir\") pod \"be07924b-4a72-4f8f-8ff0-e4e71854b388\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " Oct 30 06:23:20.486668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486475 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm\") pod \"be07924b-4a72-4f8f-8ff0-e4e71854b388\" (UID: \"be07924b-4a72-4f8f-8ff0-e4e71854b388\") " Oct 30 06:23:20.486668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486492 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home\") pod \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\" (UID: \"29a9f246-2ac1-4eb4-a230-8d028068b3dd\") " Oct 30 06:23:20.486862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.486830 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir" pod "be07924b-4a72-4f8f-8ff0-e4e71854b388" (UID: "be07924b-4a72-4f8f-8ff0-e4e71854b388"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.487083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.487045 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.487083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.487041 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home" pod "29a9f246-2ac1-4eb4-a230-8d028068b3dd" (UID: "29a9f246-2ac1-4eb4-a230-8d028068b3dd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.487655 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.487624 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache" pod "be07924b-4a72-4f8f-8ff0-e4e71854b388" (UID: "be07924b-4a72-4f8f-8ff0-e4e71854b388"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.487852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.487829 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache" pod "29a9f246-2ac1-4eb4-a230-8d028068b3dd" (UID: "29a9f246-2ac1-4eb4-a230-8d028068b3dd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.488015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.487988 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir" pod "29a9f246-2ac1-4eb4-a230-8d028068b3dd" (UID: "29a9f246-2ac1-4eb4-a230-8d028068b3dd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.489406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.489352 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs" pod "29a9f246-2ac1-4eb4-a230-8d028068b3dd" (UID: "29a9f246-2ac1-4eb4-a230-8d028068b3dd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:20.491151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.491125 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm" pod "be07924b-4a72-4f8f-8ff0-e4e71854b388" (UID: "be07924b-4a72-4f8f-8ff0-e4e71854b388"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.491239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.491202 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm" pod "29a9f246-2ac1-4eb4-a230-8d028068b3dd" (UID: "29a9f246-2ac1-4eb4-a230-8d028068b3dd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.491299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.491281 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn" pod "29a9f246-2ac1-4eb4-a230-8d028068b3dd" (UID: "29a9f246-2ac1-4eb4-a230-8d028068b3dd"). InnerVolumeSpecName "kube-api-access-vpxkn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:20.492992 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.492969 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home" pod "be07924b-4a72-4f8f-8ff0-e4e71854b388" (UID: "be07924b-4a72-4f8f-8ff0-e4e71854b388"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:20.494581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.494556 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs" pod "be07924b-4a72-4f8f-8ff0-e4e71854b388" (UID: "be07924b-4a72-4f8f-8ff0-e4e71854b388"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:20.494691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.494666 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df" pod "be07924b-4a72-4f8f-8ff0-e4e71854b388" (UID: "be07924b-4a72-4f8f-8ff0-e4e71854b388"). InnerVolumeSpecName "kube-api-access-dm5df". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589003 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm5df\" (UniqueName: \"kubernetes.io/projected/be07924b-4a72-4f8f-8ff0-e4e71854b388-kube-api-access-dm5df\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589037 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589050 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589061 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589072 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589084 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589095 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/be07924b-4a72-4f8f-8ff0-e4e71854b388-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589105 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589117 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vpxkn\" (UniqueName: \"kubernetes.io/projected/29a9f246-2ac1-4eb4-a230-8d028068b3dd-kube-api-access-vpxkn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589129 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/be07924b-4a72-4f8f-8ff0-e4e71854b388-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.589617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.589140 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/29a9f246-2ac1-4eb4-a230-8d028068b3dd-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:20.710630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.710596 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz"] Oct 30 06:23:20.711077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711064 5565 cpu_manager.go:414] "containerMap: removing container" podUID="54406b29-9815-42e7-afdb-78ca593fd477" containerName="revoke-keys" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711078 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54406b29-9815-42e7-afdb-78ca593fd477" containerName="revoke-keys" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711090 5565 cpu_manager.go:414] "containerMap: removing container" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711095 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711115 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerName="payload-pre-processing" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711121 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerName="payload-pre-processing" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711129 5565 cpu_manager.go:414] "containerMap: removing container" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" Oct 30 06:23:20.711136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711133 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" Oct 30 06:23:20.711411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711206 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="54406b29-9815-42e7-afdb-78ca593fd477" containerName="revoke-keys" Oct 30 06:23:20.711411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711217 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" Oct 30 06:23:20.711411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711226 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d17330c-7e57-4aa1-b3b2-59301813307f" containerName="payload-pre-processing" Oct 30 06:23:20.711411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.711232 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" containerName="main" Oct 30 06:23:20.715613 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.715593 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.735272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.735249 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz"] Oct 30 06:23:20.790824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.790787 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.791036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.791015 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.791101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.791090 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gf4cw\" (UniqueName: \"kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.802258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.802233 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:20.802523 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.802507 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:23:20.891818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.891747 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.891818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.891805 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gf4cw\" (UniqueName: \"kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.891993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.891872 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.892493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.892471 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.894062 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.894045 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:20.916945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:20.916920 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gf4cw\" (UniqueName: \"kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:21.028932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.028908 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:21.234234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.234144 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 06:23:21.369937 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.369911 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz"] Oct 30 06:23:21.372058 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:23:21.372028 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d119469_371a_408e_8ea9_5ca543af9e52.slice/crio-403dd567f954b21d64e10b57a1a20ec6c2ee9afd4f24bc084b2fc8628243c689 WatchSource:0}: Error finding container 403dd567f954b21d64e10b57a1a20ec6c2ee9afd4f24bc084b2fc8628243c689: Status 404 returned error can't find the container with id 403dd567f954b21d64e10b57a1a20ec6c2ee9afd4f24bc084b2fc8628243c689 Oct 30 06:23:21.376933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.376909 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" Oct 30 06:23:21.377044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.376905 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p" event={"ID":"be07924b-4a72-4f8f-8ff0-e4e71854b388","Type":"ContainerDied","Data":"0b16a6944dabcd421d73b18f265a60dd28cf37306e5b7d7fa69c340e8df41866"} Oct 30 06:23:21.377044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.377031 5565 scope.go:122] "RemoveContainer" containerID="9e17c784f73ada8f933c48ce9e50b7a8c9bd8761d701982483f74419d8afae98" Oct 30 06:23:21.378666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.378641 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" Oct 30 06:23:21.378774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.378686 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz" event={"ID":"29a9f246-2ac1-4eb4-a230-8d028068b3dd","Type":"ContainerDied","Data":"a514d820fe36aa4250ca1de34e58b9353471ef7dccd30e425488e57b2f6a8904"} Oct 30 06:23:21.389305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.389287 5565 scope.go:122] "RemoveContainer" containerID="12f1d15106a4c3ca7665857e14f5894e7384012b31f6234d27b886798118ade1" Oct 30 06:23:21.459453 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.459424 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p"] Oct 30 06:23:21.463921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.463899 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/premium-w3-fffaa8-kserve-c7b746df9-6654p"] Oct 30 06:23:21.503132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.503109 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz"] Oct 30 06:23:21.507779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.507758 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-fffaa8/distinct-2-w3-fffaa8-kserve-896d78b5c-tw5pz"] Oct 30 06:23:21.566765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.566731 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a9f246-2ac1-4eb4-a230-8d028068b3dd" path="/var/lib/kubelet/pods/29a9f246-2ac1-4eb4-a230-8d028068b3dd/volumes" Oct 30 06:23:21.567346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:21.567326 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be07924b-4a72-4f8f-8ff0-e4e71854b388" path="/var/lib/kubelet/pods/be07924b-4a72-4f8f-8ff0-e4e71854b388/volumes" Oct 30 06:23:22.387315 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:22.387285 5565 generic.go:356] "Generic (PLEG): container finished" podID="9d119469-371a-408e-8ea9-5ca543af9e52" containerID="ab09e7d2488e58909661e7b79fdba4a3b221726b649c5b4e483879c701958844" exitCode=0 Oct 30 06:23:22.387663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:22.387369 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" event={"ID":"9d119469-371a-408e-8ea9-5ca543af9e52","Type":"ContainerDied","Data":"ab09e7d2488e58909661e7b79fdba4a3b221726b649c5b4e483879c701958844"} Oct 30 06:23:22.387663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:22.387448 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" event={"ID":"9d119469-371a-408e-8ea9-5ca543af9e52","Type":"ContainerStarted","Data":"403dd567f954b21d64e10b57a1a20ec6c2ee9afd4f24bc084b2fc8628243c689"} Oct 30 06:23:23.540027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.540004 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:23.625279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.625252 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw\" (UniqueName: \"kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw\") pod \"9d119469-371a-408e-8ea9-5ca543af9e52\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " Oct 30 06:23:23.625458 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.625310 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca\") pod \"9d119469-371a-408e-8ea9-5ca543af9e52\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " Oct 30 06:23:23.625458 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.625338 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca\") pod \"9d119469-371a-408e-8ea9-5ca543af9e52\" (UID: \"9d119469-371a-408e-8ea9-5ca543af9e52\") " Oct 30 06:23:23.625811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.625782 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca" pod "9d119469-371a-408e-8ea9-5ca543af9e52" (UID: "9d119469-371a-408e-8ea9-5ca543af9e52"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:23.625935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.625910 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:23.627345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.627313 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca" pod "9d119469-371a-408e-8ea9-5ca543af9e52" (UID: "9d119469-371a-408e-8ea9-5ca543af9e52"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:23.627505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.627479 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw" pod "9d119469-371a-408e-8ea9-5ca543af9e52" (UID: "9d119469-371a-408e-8ea9-5ca543af9e52"). InnerVolumeSpecName "kube-api-access-gf4cw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:23.727054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.726966 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gf4cw\" (UniqueName: \"kubernetes.io/projected/9d119469-371a-408e-8ea9-5ca543af9e52-kube-api-access-gf4cw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:23.727054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:23.726997 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d119469-371a-408e-8ea9-5ca543af9e52-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:24.402572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:24.402515 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" Oct 30 06:23:24.402572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:24.402560 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz" event={"ID":"9d119469-371a-408e-8ea9-5ca543af9e52","Type":"ContainerDied","Data":"403dd567f954b21d64e10b57a1a20ec6c2ee9afd4f24bc084b2fc8628243c689"} Oct 30 06:23:24.402786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:24.402611 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="403dd567f954b21d64e10b57a1a20ec6c2ee9afd4f24bc084b2fc8628243c689" Oct 30 06:23:25.411406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.411365 5565 generic.go:356] "Generic (PLEG): container finished" podID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerID="7dc99a69d1a41e71c2c0959b848c7f06e5e9201e4ae2e735671aaadb3110d6d6" exitCode=0 Oct 30 06:23:25.411865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.411416 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" event={"ID":"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f","Type":"ContainerDied","Data":"7dc99a69d1a41e71c2c0959b848c7f06e5e9201e4ae2e735671aaadb3110d6d6"} Oct 30 06:23:25.411865 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.411477 5565 scope.go:122] "RemoveContainer" containerID="9c2e05b232a3aed282498bae4c364ca15ab565a32e7d53f643cc3e175f93ea4c" Oct 30 06:23:25.509533 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.509500 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 06:23:25.654713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.654682 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.654899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.654724 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.654899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.654806 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.654899 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.654886 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.655085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.654924 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.655085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.654984 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.655085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655023 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.655085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655057 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.655085 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655083 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data\") pod \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\" (UID: \"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f\") " Oct 30 06:23:25.655367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655094 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.655367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655181 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.655367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655292 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:25.655579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655443 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.655579 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655453 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.655716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655615 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.655716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655644 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.655716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655664 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.655716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655677 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.655716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.655690 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.657384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.657353 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:25.657645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.657624 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:25.657645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.657634 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:25.657790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.657772 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz" pod "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" (UID: "4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f"). InnerVolumeSpecName "kube-api-access-hvbvz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:25.756665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.756636 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.756665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.756660 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.756665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.756670 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvbvz\" (UniqueName: \"kubernetes.io/projected/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-kube-api-access-hvbvz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.756871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.756679 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:25.820558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.820512 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6"] Oct 30 06:23:25.821041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821022 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9d119469-371a-408e-8ea9-5ca543af9e52" containerName="revoke-keys" Oct 30 06:23:25.821147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821043 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d119469-371a-408e-8ea9-5ca543af9e52" containerName="revoke-keys" Oct 30 06:23:25.821147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821087 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" Oct 30 06:23:25.821147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821096 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" Oct 30 06:23:25.821147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821122 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" Oct 30 06:23:25.821147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821129 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" Oct 30 06:23:25.821420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821259 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" Oct 30 06:23:25.821420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821274 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" containerName="istio-proxy" Oct 30 06:23:25.821420 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.821289 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d119469-371a-408e-8ea9-5ca543af9e52" containerName="revoke-keys" Oct 30 06:23:25.824960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.824940 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:25.845868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.845835 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6"] Oct 30 06:23:25.900217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.900191 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:23:25.900367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.900244 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:25.958951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.958920 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px6r4\" (UniqueName: \"kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:25.959100 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.959081 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:25.959186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:25.959130 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.060339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.060276 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.060339 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.060311 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.060512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.060407 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px6r4\" (UniqueName: \"kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.061035 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.061015 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.062669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.062648 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.086717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.086698 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px6r4\" (UniqueName: \"kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.138502 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.138480 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:26.419384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.419310 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" Oct 30 06:23:26.419384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.419337 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx" event={"ID":"4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f","Type":"ContainerDied","Data":"58c8b7e685aa39f52f35f2e240db88d0835992d8ee072a48f3ac6d00469a264f"} Oct 30 06:23:26.419911 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.419389 5565 scope.go:122] "RemoveContainer" containerID="7dc99a69d1a41e71c2c0959b848c7f06e5e9201e4ae2e735671aaadb3110d6d6" Oct 30 06:23:26.441161 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:23:26.441130 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae534f52_c075_479e_b87f_dec20b6e69d7.slice/crio-8e9276348867537961d38a5536d7a5be17511f3dfea1250a8fca96eee3a8af20 WatchSource:0}: Error finding container 8e9276348867537961d38a5536d7a5be17511f3dfea1250a8fca96eee3a8af20: Status 404 returned error can't find the container with id 8e9276348867537961d38a5536d7a5be17511f3dfea1250a8fca96eee3a8af20 Oct 30 06:23:26.452436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.452411 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6"] Oct 30 06:23:26.477481 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.477456 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx"] Oct 30 06:23:26.484623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.484592 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-3abe42-openshift-default-8447595dc-tgmnx"] Oct 30 06:23:26.799177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.799150 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx"] Oct 30 06:23:26.803036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.803018 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.832823 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.832794 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx"] Oct 30 06:23:26.869678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.869645 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.869839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.869761 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jljft\" (UniqueName: \"kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.869839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.869793 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.970831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.970750 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.970948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.970842 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.970948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.970912 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jljft\" (UniqueName: \"kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.971473 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.971452 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.973136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.973117 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:26.996176 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:26.996144 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jljft\" (UniqueName: \"kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft\") pod \"maas-api-revoke-sub-0a510e20-2tztx\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:27.117665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.117627 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:27.423331 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.423299 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx"] Oct 30 06:23:27.430951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.430921 5565 generic.go:356] "Generic (PLEG): container finished" podID="ae534f52-c075-479e-b87f-dec20b6e69d7" containerID="514e41d9dbc5e182645f812b51d1f4cc3ababb3b3404c3bb2e7c09095cbee4aa" exitCode=0 Oct 30 06:23:27.431132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.431009 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" event={"ID":"ae534f52-c075-479e-b87f-dec20b6e69d7","Type":"ContainerDied","Data":"514e41d9dbc5e182645f812b51d1f4cc3ababb3b3404c3bb2e7c09095cbee4aa"} Oct 30 06:23:27.431132 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.431072 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" event={"ID":"ae534f52-c075-479e-b87f-dec20b6e69d7","Type":"ContainerStarted","Data":"8e9276348867537961d38a5536d7a5be17511f3dfea1250a8fca96eee3a8af20"} Oct 30 06:23:27.434503 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:23:27.434479 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod415f8bc5_30d5_45c0_a25e_648b7b3cfc26.slice/crio-6d1f3e5de86b668de088bb51913572052b5ee45831aa4874203074cdbb2df590 WatchSource:0}: Error finding container 6d1f3e5de86b668de088bb51913572052b5ee45831aa4874203074cdbb2df590: Status 404 returned error can't find the container with id 6d1f3e5de86b668de088bb51913572052b5ee45831aa4874203074cdbb2df590 Oct 30 06:23:27.567069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.567045 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f" path="/var/lib/kubelet/pods/4bee2e5b-a4b9-462f-bf9d-8c4cc1ffcf9f/volumes" Oct 30 06:23:27.727854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.727821 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj"] Oct 30 06:23:27.729976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:27.729955 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-720e9c-5lbqj"] Oct 30 06:23:28.439684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.439645 5565 generic.go:356] "Generic (PLEG): container finished" podID="415f8bc5-30d5-45c0-a25e-648b7b3cfc26" containerID="9dfbed8469979e7a6b14b82d89ed5b58729c88ab1e37594c980a458236a8f2c8" exitCode=0 Oct 30 06:23:28.440084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.439728 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" event={"ID":"415f8bc5-30d5-45c0-a25e-648b7b3cfc26","Type":"ContainerDied","Data":"9dfbed8469979e7a6b14b82d89ed5b58729c88ab1e37594c980a458236a8f2c8"} Oct 30 06:23:28.440084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.439780 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" event={"ID":"415f8bc5-30d5-45c0-a25e-648b7b3cfc26","Type":"ContainerStarted","Data":"6d1f3e5de86b668de088bb51913572052b5ee45831aa4874203074cdbb2df590"} Oct 30 06:23:28.592119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.592093 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:28.692908 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.692878 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4\" (UniqueName: \"kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4\") pod \"ae534f52-c075-479e-b87f-dec20b6e69d7\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " Oct 30 06:23:28.693089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.692992 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca\") pod \"ae534f52-c075-479e-b87f-dec20b6e69d7\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " Oct 30 06:23:28.693089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.693071 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca\") pod \"ae534f52-c075-479e-b87f-dec20b6e69d7\" (UID: \"ae534f52-c075-479e-b87f-dec20b6e69d7\") " Oct 30 06:23:28.693422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.693400 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca" pod "ae534f52-c075-479e-b87f-dec20b6e69d7" (UID: "ae534f52-c075-479e-b87f-dec20b6e69d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:28.694920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.694897 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca" pod "ae534f52-c075-479e-b87f-dec20b6e69d7" (UID: "ae534f52-c075-479e-b87f-dec20b6e69d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:28.695015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.694938 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4" pod "ae534f52-c075-479e-b87f-dec20b6e69d7" (UID: "ae534f52-c075-479e-b87f-dec20b6e69d7"). InnerVolumeSpecName "kube-api-access-px6r4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:28.793968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.793939 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px6r4\" (UniqueName: \"kubernetes.io/projected/ae534f52-c075-479e-b87f-dec20b6e69d7-kube-api-access-px6r4\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:28.793968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.793963 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:28.794117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:28.793974 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ae534f52-c075-479e-b87f-dec20b6e69d7-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:29.447783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.447753 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" Oct 30 06:23:29.448227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.447787 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6" event={"ID":"ae534f52-c075-479e-b87f-dec20b6e69d7","Type":"ContainerDied","Data":"8e9276348867537961d38a5536d7a5be17511f3dfea1250a8fca96eee3a8af20"} Oct 30 06:23:29.448227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.447842 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8e9276348867537961d38a5536d7a5be17511f3dfea1250a8fca96eee3a8af20" Oct 30 06:23:29.567517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.567485 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54406b29-9815-42e7-afdb-78ca593fd477" path="/var/lib/kubelet/pods/54406b29-9815-42e7-afdb-78ca593fd477/volumes" Oct 30 06:23:29.606640 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.606609 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:29.703711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.703640 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca\") pod \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " Oct 30 06:23:29.703711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.703673 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft\" (UniqueName: \"kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft\") pod \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " Oct 30 06:23:29.703711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.703693 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca\") pod \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\" (UID: \"415f8bc5-30d5-45c0-a25e-648b7b3cfc26\") " Oct 30 06:23:29.704078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.704038 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca" pod "415f8bc5-30d5-45c0-a25e-648b7b3cfc26" (UID: "415f8bc5-30d5-45c0-a25e-648b7b3cfc26"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:29.704208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.704193 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:29.705828 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.705806 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca" pod "415f8bc5-30d5-45c0-a25e-648b7b3cfc26" (UID: "415f8bc5-30d5-45c0-a25e-648b7b3cfc26"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:29.705925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.705902 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft" pod "415f8bc5-30d5-45c0-a25e-648b7b3cfc26" (UID: "415f8bc5-30d5-45c0-a25e-648b7b3cfc26"). InnerVolumeSpecName "kube-api-access-jljft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:29.805576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.805553 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jljft\" (UniqueName: \"kubernetes.io/projected/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-kube-api-access-jljft\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:29.805576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:29.805577 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/415f8bc5-30d5-45c0-a25e-648b7b3cfc26-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:30.456236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:30.456206 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" Oct 30 06:23:30.456659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:30.456239 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx" event={"ID":"415f8bc5-30d5-45c0-a25e-648b7b3cfc26","Type":"ContainerDied","Data":"6d1f3e5de86b668de088bb51913572052b5ee45831aa4874203074cdbb2df590"} Oct 30 06:23:30.456659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:30.456289 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d1f3e5de86b668de088bb51913572052b5ee45831aa4874203074cdbb2df590" Oct 30 06:23:35.111389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.111356 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5"] Oct 30 06:23:35.111903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.111884 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ae534f52-c075-479e-b87f-dec20b6e69d7" containerName="revoke-keys" Oct 30 06:23:35.111961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.111906 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae534f52-c075-479e-b87f-dec20b6e69d7" containerName="revoke-keys" Oct 30 06:23:35.111961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.111920 5565 cpu_manager.go:414] "containerMap: removing container" podUID="415f8bc5-30d5-45c0-a25e-648b7b3cfc26" containerName="revoke-keys" Oct 30 06:23:35.111961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.111926 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="415f8bc5-30d5-45c0-a25e-648b7b3cfc26" containerName="revoke-keys" Oct 30 06:23:35.112063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.112025 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae534f52-c075-479e-b87f-dec20b6e69d7" containerName="revoke-keys" Oct 30 06:23:35.112063 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.112037 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="415f8bc5-30d5-45c0-a25e-648b7b3cfc26" containerName="revoke-keys" Oct 30 06:23:35.115724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.115705 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.136852 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.136803 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5"] Oct 30 06:23:35.172944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.172916 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:23:35.173088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.173009 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:23:35.261293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.261263 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.261421 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.261326 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.261463 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.261438 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpnpc\" (UniqueName: \"kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.362424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.362347 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.362673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.362422 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.362673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.362485 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpnpc\" (UniqueName: \"kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.363084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.363062 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.364796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.364776 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.387326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.387298 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpnpc\" (UniqueName: \"kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.429808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.429787 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:35.750783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:35.750755 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5"] Oct 30 06:23:35.753321 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:23:35.753292 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fb6a802_ecb7_4eb6_a861_b3a44d9a08a0.slice/crio-cfb20dd65bf95f68296c7856d8bbf6c7a2534e1cb2648f42b8c6fb10aa4fff84 WatchSource:0}: Error finding container cfb20dd65bf95f68296c7856d8bbf6c7a2534e1cb2648f42b8c6fb10aa4fff84: Status 404 returned error can't find the container with id cfb20dd65bf95f68296c7856d8bbf6c7a2534e1cb2648f42b8c6fb10aa4fff84 Oct 30 06:23:36.499310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:36.499219 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" event={"ID":"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0","Type":"ContainerStarted","Data":"1c256d0f4ef47616a8d710e23aee5378be2bd7e3290c05723bfe88065420fef3"} Oct 30 06:23:36.499310 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:36.499290 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" event={"ID":"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0","Type":"ContainerStarted","Data":"cfb20dd65bf95f68296c7856d8bbf6c7a2534e1cb2648f42b8c6fb10aa4fff84"} Oct 30 06:23:36.544676 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:36.544624 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" podStartSLOduration=1.54461118 podStartE2EDuration="1.54461118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:23:36.542628929 +0000 UTC m=+3123.118315984" watchObservedRunningTime="2026-10-30 06:23:36.54461118 +0000 UTC m=+3123.120298241" Oct 30 06:23:37.091808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:37.091776 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:23:37.506953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:37.506915 5565 generic.go:356] "Generic (PLEG): container finished" podID="1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" containerID="1c256d0f4ef47616a8d710e23aee5378be2bd7e3290c05723bfe88065420fef3" exitCode=0 Oct 30 06:23:37.507355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:37.506995 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" event={"ID":"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0","Type":"ContainerDied","Data":"1c256d0f4ef47616a8d710e23aee5378be2bd7e3290c05723bfe88065420fef3"} Oct 30 06:23:38.655257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.655229 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:38.802825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.802751 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca\") pod \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " Oct 30 06:23:38.802825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.802814 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca\") pod \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " Oct 30 06:23:38.803019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.802864 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc\" (UniqueName: \"kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc\") pod \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\" (UID: \"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0\") " Oct 30 06:23:38.803092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.803071 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca" pod "1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" (UID: "1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:38.803226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.803213 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:38.804961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.804934 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc" pod "1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" (UID: "1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0"). InnerVolumeSpecName "kube-api-access-jpnpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:38.805072 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.805035 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca" pod "1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" (UID: "1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:38.904171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.904143 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:38.904171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:38.904169 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpnpc\" (UniqueName: \"kubernetes.io/projected/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0-kube-api-access-jpnpc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:39.520633 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:39.520595 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" event={"ID":"1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0","Type":"ContainerDied","Data":"cfb20dd65bf95f68296c7856d8bbf6c7a2534e1cb2648f42b8c6fb10aa4fff84"} Oct 30 06:23:39.520759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:39.520628 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5" Oct 30 06:23:39.520759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:39.520661 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb20dd65bf95f68296c7856d8bbf6c7a2534e1cb2648f42b8c6fb10aa4fff84" Oct 30 06:23:41.792105 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:41.792070 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz"] Oct 30 06:23:41.814498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:41.814464 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-452a59-rtlkz"] Oct 30 06:23:43.565903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:43.565869 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d119469-371a-408e-8ea9-5ca543af9e52" path="/var/lib/kubelet/pods/9d119469-371a-408e-8ea9-5ca543af9e52/volumes" Oct 30 06:23:44.122098 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:44.122060 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6"] Oct 30 06:23:44.145351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:44.142662 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-c2be21-5c6h6"] Oct 30 06:23:45.568693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:45.568661 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae534f52-c075-479e-b87f-dec20b6e69d7" path="/var/lib/kubelet/pods/ae534f52-c075-479e-b87f-dec20b6e69d7/volumes" Oct 30 06:23:47.533535 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:47.533479 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:23:49.251902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:49.251865 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr"] Oct 30 06:23:49.252409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:49.252103 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" containerID="cri-o://3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64" gracePeriod=30 Oct 30 06:23:49.910101 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:49.910056 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv"] Oct 30 06:23:49.910360 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:49.910323 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" containerID="cri-o://b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a" gracePeriod=30 Oct 30 06:23:53.073460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.073427 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz"] Oct 30 06:23:53.074043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.073679 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" podUID="9412d483-a3c7-4bb4-89bf-d620bb74c382" containerName="maas-api" containerID="cri-o://f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b" gracePeriod=30 Oct 30 06:23:53.359596 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.355164 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:23:53.455854 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.455814 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls\") pod \"9412d483-a3c7-4bb4-89bf-d620bb74c382\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " Oct 30 06:23:53.456046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.455915 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca\") pod \"9412d483-a3c7-4bb4-89bf-d620bb74c382\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " Oct 30 06:23:53.456046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.455990 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc\" (UniqueName: \"kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc\") pod \"9412d483-a3c7-4bb4-89bf-d620bb74c382\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " Oct 30 06:23:53.456046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.456029 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs\") pod \"9412d483-a3c7-4bb4-89bf-d620bb74c382\" (UID: \"9412d483-a3c7-4bb4-89bf-d620bb74c382\") " Oct 30 06:23:53.456337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.456296 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca" pod "9412d483-a3c7-4bb4-89bf-d620bb74c382" (UID: "9412d483-a3c7-4bb4-89bf-d620bb74c382"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:53.458015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.457977 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls" pod "9412d483-a3c7-4bb4-89bf-d620bb74c382" (UID: "9412d483-a3c7-4bb4-89bf-d620bb74c382"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:53.458129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.458075 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs" pod "9412d483-a3c7-4bb4-89bf-d620bb74c382" (UID: "9412d483-a3c7-4bb4-89bf-d620bb74c382"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:23:53.458129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.458122 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc" pod "9412d483-a3c7-4bb4-89bf-d620bb74c382" (UID: "9412d483-a3c7-4bb4-89bf-d620bb74c382"). InnerVolumeSpecName "kube-api-access-g4grc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:53.557346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.557314 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9412d483-a3c7-4bb4-89bf-d620bb74c382-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.557346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.557342 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4grc\" (UniqueName: \"kubernetes.io/projected/9412d483-a3c7-4bb4-89bf-d620bb74c382-kube-api-access-g4grc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.557346 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.557355 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.557621 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.557364 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9412d483-a3c7-4bb4-89bf-d620bb74c382-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:53.615928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.615857 5565 generic.go:356] "Generic (PLEG): container finished" podID="9412d483-a3c7-4bb4-89bf-d620bb74c382" containerID="f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b" exitCode=0 Oct 30 06:23:53.615928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.615897 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" Oct 30 06:23:53.615928 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.615910 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" event={"ID":"9412d483-a3c7-4bb4-89bf-d620bb74c382","Type":"ContainerDied","Data":"f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b"} Oct 30 06:23:53.616154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.615959 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz" event={"ID":"9412d483-a3c7-4bb4-89bf-d620bb74c382","Type":"ContainerDied","Data":"9fd73a66e0027ed6283f5cd6f099230e3c483aa6eeca3102d0fc0274686ee6ee"} Oct 30 06:23:53.616154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.615991 5565 scope.go:122] "RemoveContainer" containerID="f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b" Oct 30 06:23:53.625933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.625913 5565 scope.go:122] "RemoveContainer" containerID="f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b" Oct 30 06:23:53.626214 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:53.626195 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b\": container with ID starting with f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b not found: ID does not exist" containerID="f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b" Oct 30 06:23:53.626302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.626222 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b"} err="failed to get container status \"f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b\": rpc error: code = NotFound desc = could not find container \"f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b\": container with ID starting with f0740f47b3f4e5328b7e3ac5496384ee72f4a80b605d44df824b1495ec59033b not found: ID does not exist" Oct 30 06:23:53.672404 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.672378 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz"] Oct 30 06:23:53.686465 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.686440 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w6-95498c-58499f745d-ptfpz"] Oct 30 06:23:53.946693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.946611 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5"] Oct 30 06:23:53.948663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:53.948633 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-95498c-mzgc5"] Oct 30 06:23:54.523369 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.523315 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 06:23:54.567799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.567772 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.567932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.567838 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.567932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.567870 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.567932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.567898 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.568114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.567933 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.568114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.567975 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.568114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568009 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.568274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568159 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.568274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568188 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.568274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568200 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.568274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568260 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo\") pod \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\" (UID: \"9b56a291-a225-48d1-8af8-f331c2eb4ae4\") " Oct 30 06:23:54.568504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568333 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.568504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568377 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:54.568504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568397 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.568750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568728 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.568804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568758 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.568804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568774 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.568804 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568788 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.568921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.568803 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.570020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.569992 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:54.570215 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.570199 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:54.570632 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.570609 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:54.570731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.570635 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488" pod "9b56a291-a225-48d1-8af8-f331c2eb4ae4" (UID: "9b56a291-a225-48d1-8af8-f331c2eb4ae4"). InnerVolumeSpecName "kube-api-access-9w488". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:54.637033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.636993 5565 generic.go:356] "Generic (PLEG): container finished" podID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerID="3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64" exitCode=0 Oct 30 06:23:54.637204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.637042 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" event={"ID":"9b56a291-a225-48d1-8af8-f331c2eb4ae4","Type":"ContainerDied","Data":"3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64"} Oct 30 06:23:54.637204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.637056 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" Oct 30 06:23:54.637204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.637084 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr" event={"ID":"9b56a291-a225-48d1-8af8-f331c2eb4ae4","Type":"ContainerDied","Data":"b5d1a58480e72d2b838a58cf1190014822f733cadcdd8d8a3bcd8f0e544e2464"} Oct 30 06:23:54.637204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.637106 5565 scope.go:122] "RemoveContainer" containerID="3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64" Oct 30 06:23:54.648269 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.648252 5565 scope.go:122] "RemoveContainer" containerID="8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954" Oct 30 06:23:54.659434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.659414 5565 scope.go:122] "RemoveContainer" containerID="3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64" Oct 30 06:23:54.659712 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:54.659684 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64\": container with ID starting with 3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64 not found: ID does not exist" containerID="3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64" Oct 30 06:23:54.659803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.659722 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64"} err="failed to get container status \"3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64\": rpc error: code = NotFound desc = could not find container \"3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64\": container with ID starting with 3cf5c4781259c8345120d9d139ce330d7728f1bdd08a76f015e261544389fe64 not found: ID does not exist" Oct 30 06:23:54.659803 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.659744 5565 scope.go:122] "RemoveContainer" containerID="8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954" Oct 30 06:23:54.659994 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:54.659978 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954\": container with ID starting with 8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954 not found: ID does not exist" containerID="8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954" Oct 30 06:23:54.660031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.659999 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954"} err="failed to get container status \"8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954\": rpc error: code = NotFound desc = could not find container \"8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954\": container with ID starting with 8813a0e365ffba72d720a1a062f22da17b11d2e4cd9b8125da2d408f8b5ed954 not found: ID does not exist" Oct 30 06:23:54.669610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.669591 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.669610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.669611 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.669729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.669620 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w488\" (UniqueName: \"kubernetes.io/projected/9b56a291-a225-48d1-8af8-f331c2eb4ae4-kube-api-access-9w488\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.669729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.669629 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b56a291-a225-48d1-8af8-f331c2eb4ae4-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:54.694675 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.694651 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr"] Oct 30 06:23:54.704040 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:54.704018 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-fffaa8-openshift-default-5dfff7777f-2nrkr"] Oct 30 06:23:55.152494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.152470 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 06:23:55.174199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174172 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174211 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174241 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174284 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174319 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174349 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174391 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174425 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174615 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174521 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg\") pod \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\" (UID: \"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0\") " Oct 30 06:23:55.174912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174640 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.174912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174775 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.174912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174789 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.174912 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.174902 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.175149 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.175104 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.175149 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.175122 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.175149 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.175136 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.175149 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.175149 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.175382 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.175216 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:23:55.176451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.176422 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:23:55.176772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.176753 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "kube-api-access-h68cg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.176841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.176808 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:23:55.176841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.176818 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token" pod "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" (UID: "482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:23:55.276592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.276568 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.276592 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.276591 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.276770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.276605 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.276770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.276616 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.276770 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.276624 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h68cg\" (UniqueName: \"kubernetes.io/projected/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0-kube-api-access-h68cg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:23:55.369696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.369664 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv"] Oct 30 06:23:55.566584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.566484 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" path="/var/lib/kubelet/pods/1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0/volumes" Oct 30 06:23:55.567147 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.567123 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9412d483-a3c7-4bb4-89bf-d620bb74c382" path="/var/lib/kubelet/pods/9412d483-a3c7-4bb4-89bf-d620bb74c382/volumes" Oct 30 06:23:55.567630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.567602 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" path="/var/lib/kubelet/pods/9b56a291-a225-48d1-8af8-f331c2eb4ae4/volumes" Oct 30 06:23:55.653351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.653320 5565 generic.go:356] "Generic (PLEG): container finished" podID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerID="b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a" exitCode=0 Oct 30 06:23:55.653489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.653375 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" Oct 30 06:23:55.653489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.653400 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" event={"ID":"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0","Type":"ContainerDied","Data":"b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a"} Oct 30 06:23:55.653489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.653470 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv" event={"ID":"482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0","Type":"ContainerDied","Data":"6a077ac003f26bcbf1f98f25b15b2b243799b8981d3168b1121c7bd7fbd32ae8"} Oct 30 06:23:55.653635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.653492 5565 scope.go:122] "RemoveContainer" containerID="b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a" Oct 30 06:23:55.663370 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.663348 5565 scope.go:122] "RemoveContainer" containerID="aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4" Oct 30 06:23:55.672880 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.672858 5565 scope.go:122] "RemoveContainer" containerID="b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a" Oct 30 06:23:55.673150 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:55.673127 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a\": container with ID starting with b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a not found: ID does not exist" containerID="b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a" Oct 30 06:23:55.673254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.673156 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a"} err="failed to get container status \"b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a\": rpc error: code = NotFound desc = could not find container \"b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a\": container with ID starting with b8b6eea1fcbabccea9ff9b2d9e806430a3801558b529eef74384ff705d79f29a not found: ID does not exist" Oct 30 06:23:55.673254 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.673173 5565 scope.go:122] "RemoveContainer" containerID="aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4" Oct 30 06:23:55.673443 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:23:55.673420 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4\": container with ID starting with aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4 not found: ID does not exist" containerID="aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4" Oct 30 06:23:55.673494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.673451 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4"} err="failed to get container status \"aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4\": rpc error: code = NotFound desc = could not find container \"aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4\": container with ID starting with aa1334c59ca9483dc4dbebfe40f5f0d377829996ab9e2992734cd2b116f156c4 not found: ID does not exist" Oct 30 06:23:55.711027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.710998 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv"] Oct 30 06:23:55.724793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:55.724769 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-452a59-openshift-default-788959ddc-gp6dv"] Oct 30 06:23:57.566861 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:57.566823 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" path="/var/lib/kubelet/pods/482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0/volumes" Oct 30 06:23:59.364068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:23:59.364028 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:24:01.028074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:01.028040 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd"] Oct 30 06:24:01.060541 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:01.060509 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds"] Oct 30 06:24:02.583781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.583745 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv"] Oct 30 06:24:02.584189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584176 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9412d483-a3c7-4bb4-89bf-d620bb74c382" containerName="maas-api" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584190 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9412d483-a3c7-4bb4-89bf-d620bb74c382" containerName="maas-api" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584205 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584210 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584232 5565 cpu_manager.go:414] "containerMap: removing container" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584237 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584244 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" containerName="revoke-keys" Oct 30 06:24:02.584259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584249 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" containerName="revoke-keys" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584264 5565 cpu_manager.go:414] "containerMap: removing container" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584268 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584335 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584346 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584354 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584359 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fb6a802-ecb7-4eb6-a861-b3a44d9a08a0" containerName="revoke-keys" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584366 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="482cef3a-e4eb-4b46-a5e2-40ef8ead3ea0" containerName="istio-proxy" Oct 30 06:24:02.584516 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.584373 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9412d483-a3c7-4bb4-89bf-d620bb74c382" containerName="maas-api" Oct 30 06:24:02.589472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.589452 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.608553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.608503 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv"] Oct 30 06:24:02.612345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.612314 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:02.612490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.612350 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:02.620553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.620508 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:02.620925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.620907 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:24:02.656477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.656450 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.656623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.656492 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.656623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.656608 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lq2rt\" (UniqueName: \"kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.758380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.758342 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.758587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.758414 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.758587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.758466 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lq2rt\" (UniqueName: \"kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.759239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.759206 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.761156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.761132 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.783881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.783854 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lq2rt\" (UniqueName: \"kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt\") pod \"maas-api-revoke-sub-9252fa77-cf4wv\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:02.902354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:02.902272 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:03.204790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:03.204659 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv"] Oct 30 06:24:03.220518 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:24:03.220462 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39c60b43_61b5_4368_9e39_d90264f28536.slice/crio-87d7ac2ef2995bc0933ab6768cd125aa960b58d68702c35df9c5fc400b0b97d4 WatchSource:0}: Error finding container 87d7ac2ef2995bc0933ab6768cd125aa960b58d68702c35df9c5fc400b0b97d4: Status 404 returned error can't find the container with id 87d7ac2ef2995bc0933ab6768cd125aa960b58d68702c35df9c5fc400b0b97d4 Oct 30 06:24:03.703213 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:03.703177 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" event={"ID":"39c60b43-61b5-4368-9e39-d90264f28536","Type":"ContainerStarted","Data":"ed75cd69896176e16037cad184e3e0bf6cbe7578927d095fd54069eca8cb0d63"} Oct 30 06:24:03.703564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:03.703231 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" event={"ID":"39c60b43-61b5-4368-9e39-d90264f28536","Type":"ContainerStarted","Data":"87d7ac2ef2995bc0933ab6768cd125aa960b58d68702c35df9c5fc400b0b97d4"} Oct 30 06:24:03.762033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:03.761988 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" podStartSLOduration=1.7619747989999999 podStartE2EDuration="1.761974799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:03.747075606 +0000 UTC m=+3150.322762680" watchObservedRunningTime="2026-10-30 06:24:03.761974799 +0000 UTC m=+3150.337661853" Oct 30 06:24:04.712193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:04.712150 5565 generic.go:356] "Generic (PLEG): container finished" podID="39c60b43-61b5-4368-9e39-d90264f28536" containerID="ed75cd69896176e16037cad184e3e0bf6cbe7578927d095fd54069eca8cb0d63" exitCode=0 Oct 30 06:24:04.712745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:04.712237 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" event={"ID":"39c60b43-61b5-4368-9e39-d90264f28536","Type":"ContainerDied","Data":"ed75cd69896176e16037cad184e3e0bf6cbe7578927d095fd54069eca8cb0d63"} Oct 30 06:24:05.857807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.857784 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:05.992206 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.992136 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca\") pod \"39c60b43-61b5-4368-9e39-d90264f28536\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " Oct 30 06:24:05.992354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.992219 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt\" (UniqueName: \"kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt\") pod \"39c60b43-61b5-4368-9e39-d90264f28536\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " Oct 30 06:24:05.992354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.992326 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca\") pod \"39c60b43-61b5-4368-9e39-d90264f28536\" (UID: \"39c60b43-61b5-4368-9e39-d90264f28536\") " Oct 30 06:24:05.992587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.992559 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca" pod "39c60b43-61b5-4368-9e39-d90264f28536" (UID: "39c60b43-61b5-4368-9e39-d90264f28536"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:05.994304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.994278 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca" pod "39c60b43-61b5-4368-9e39-d90264f28536" (UID: "39c60b43-61b5-4368-9e39-d90264f28536"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:05.994390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:05.994299 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt" pod "39c60b43-61b5-4368-9e39-d90264f28536" (UID: "39c60b43-61b5-4368-9e39-d90264f28536"). InnerVolumeSpecName "kube-api-access-lq2rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:06.028716 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.028659 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerName="payload-processing" containerID="cri-o://9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9" gracePeriod=25 Oct 30 06:24:06.061305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.061260 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerName="payload-pre-processing" containerID="cri-o://ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13" gracePeriod=25 Oct 30 06:24:06.093332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.093307 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.093332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.093333 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lq2rt\" (UniqueName: \"kubernetes.io/projected/39c60b43-61b5-4368-9e39-d90264f28536-kube-api-access-lq2rt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.093461 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.093344 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39c60b43-61b5-4368-9e39-d90264f28536-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:06.726871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.726843 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" Oct 30 06:24:06.727027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.726874 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv" event={"ID":"39c60b43-61b5-4368-9e39-d90264f28536","Type":"ContainerDied","Data":"87d7ac2ef2995bc0933ab6768cd125aa960b58d68702c35df9c5fc400b0b97d4"} Oct 30 06:24:06.727027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.726920 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="87d7ac2ef2995bc0933ab6768cd125aa960b58d68702c35df9c5fc400b0b97d4" Oct 30 06:24:06.843906 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.843877 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp"] Oct 30 06:24:06.858287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:06.858257 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245"] Oct 30 06:24:10.390055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.390000 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" containerID="cri-o://d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7" gracePeriod=45 Oct 30 06:24:10.639025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.639000 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 06:24:10.740181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740092 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home\") pod \"3482a4c5-784b-4f8c-9e34-83429255b0b0\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " Oct 30 06:24:10.740368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740220 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache\") pod \"3482a4c5-784b-4f8c-9e34-83429255b0b0\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " Oct 30 06:24:10.740368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740264 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir\") pod \"3482a4c5-784b-4f8c-9e34-83429255b0b0\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " Oct 30 06:24:10.740368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740351 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4\" (UniqueName: \"kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4\") pod \"3482a4c5-784b-4f8c-9e34-83429255b0b0\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " Oct 30 06:24:10.740368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740353 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home" pod "3482a4c5-784b-4f8c-9e34-83429255b0b0" (UID: "3482a4c5-784b-4f8c-9e34-83429255b0b0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:10.740660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740424 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm\") pod \"3482a4c5-784b-4f8c-9e34-83429255b0b0\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " Oct 30 06:24:10.740660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740473 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache" pod "3482a4c5-784b-4f8c-9e34-83429255b0b0" (UID: "3482a4c5-784b-4f8c-9e34-83429255b0b0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:10.740660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740475 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs\" (UniqueName: \"kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs\") pod \"3482a4c5-784b-4f8c-9e34-83429255b0b0\" (UID: \"3482a4c5-784b-4f8c-9e34-83429255b0b0\") " Oct 30 06:24:10.740660 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740612 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir" pod "3482a4c5-784b-4f8c-9e34-83429255b0b0" (UID: "3482a4c5-784b-4f8c-9e34-83429255b0b0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:10.740933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740913 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.740997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740939 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.740997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.740953 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.742663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.742630 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm" pod "3482a4c5-784b-4f8c-9e34-83429255b0b0" (UID: "3482a4c5-784b-4f8c-9e34-83429255b0b0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:10.742778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.742719 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4" pod "3482a4c5-784b-4f8c-9e34-83429255b0b0" (UID: "3482a4c5-784b-4f8c-9e34-83429255b0b0"). InnerVolumeSpecName "kube-api-access-xk7x4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:10.742778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.742718 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs" pod "3482a4c5-784b-4f8c-9e34-83429255b0b0" (UID: "3482a4c5-784b-4f8c-9e34-83429255b0b0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:10.752692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.752487 5565 generic.go:356] "Generic (PLEG): container finished" podID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerID="d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7" exitCode=0 Oct 30 06:24:10.752692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.752549 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" Oct 30 06:24:10.752692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.752563 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" event={"ID":"3482a4c5-784b-4f8c-9e34-83429255b0b0","Type":"ContainerDied","Data":"d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7"} Oct 30 06:24:10.752692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.752613 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv" event={"ID":"3482a4c5-784b-4f8c-9e34-83429255b0b0","Type":"ContainerDied","Data":"5644ad36d55ac8bc3fa07e5956b53e99fe2c3d12eb090e66ca803f85106d81f0"} Oct 30 06:24:10.752692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.752635 5565 scope.go:122] "RemoveContainer" containerID="d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7" Oct 30 06:24:10.770145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.770125 5565 scope.go:122] "RemoveContainer" containerID="d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7" Oct 30 06:24:10.773105 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:10.773080 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7\": container with ID starting with d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7 not found: ID does not exist" containerID="d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7" Oct 30 06:24:10.773204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.773115 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7"} err="failed to get container status \"d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7\": rpc error: code = NotFound desc = could not find container \"d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7\": container with ID starting with d07112e34eeadb1afbf611608c19377e7daf497de6f7c5505cf4f1d29a7f09d7 not found: ID does not exist" Oct 30 06:24:10.808841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.808814 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv"] Oct 30 06:24:10.817109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.817083 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-1d8c2f/distinct-2-w1-1d8c2f-kserve-796d4d8c-mr8xv"] Oct 30 06:24:10.841935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.841901 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk7x4\" (UniqueName: \"kubernetes.io/projected/3482a4c5-784b-4f8c-9e34-83429255b0b0-kube-api-access-xk7x4\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.841935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.841926 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3482a4c5-784b-4f8c-9e34-83429255b0b0-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:10.841935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:10.841937 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3482a4c5-784b-4f8c-9e34-83429255b0b0-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:11.566570 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:11.566541 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" path="/var/lib/kubelet/pods/3482a4c5-784b-4f8c-9e34-83429255b0b0/volumes" Oct 30 06:24:11.620219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:11.620191 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:11.620367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:11.620232 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:12.392560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:12.392511 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl"] Oct 30 06:24:13.140894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:13.140851 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4"] Oct 30 06:24:13.141366 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:13.141060 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" containerID="cri-o://0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507" gracePeriod=30 Oct 30 06:24:16.431065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431029 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45"] Oct 30 06:24:16.431509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431494 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" Oct 30 06:24:16.431589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431511 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" Oct 30 06:24:16.431589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431541 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" Oct 30 06:24:16.431589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431546 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b56a291-a225-48d1-8af8-f331c2eb4ae4" containerName="istio-proxy" Oct 30 06:24:16.431589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431563 5565 cpu_manager.go:414] "containerMap: removing container" podUID="39c60b43-61b5-4368-9e39-d90264f28536" containerName="revoke-keys" Oct 30 06:24:16.431589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431568 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39c60b43-61b5-4368-9e39-d90264f28536" containerName="revoke-keys" Oct 30 06:24:16.431745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431644 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3482a4c5-784b-4f8c-9e34-83429255b0b0" containerName="main" Oct 30 06:24:16.431745 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.431659 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="39c60b43-61b5-4368-9e39-d90264f28536" containerName="revoke-keys" Oct 30 06:24:16.436287 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.436268 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.467109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.467067 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:16.467109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.467073 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:16.467109 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.467103 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:16.467474 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.467447 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:24:16.469253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.469231 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45"] Oct 30 06:24:16.593433 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.593401 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ff7pw\" (UniqueName: \"kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.593627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.593457 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.593627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.593601 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.695091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.695008 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.695252 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.695093 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ff7pw\" (UniqueName: \"kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.695338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.695261 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.695868 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.695844 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.697685 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.697662 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.719680 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.719661 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ff7pw\" (UniqueName: \"kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw\") pod \"maas-api-revoke-sub-056eac1f-62n45\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.750138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.750111 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:16.891401 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:16.891369 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:24:17.051158 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.051135 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45"] Oct 30 06:24:17.053145 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:24:17.053119 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a13018c_df26_401e_adaa_0a4bdd97e52a.slice/crio-cfd99789acc2ec1f347c278306c048a2dba6861a32c6128b8f86c1d9b0f7826c WatchSource:0}: Error finding container cfd99789acc2ec1f347c278306c048a2dba6861a32c6128b8f86c1d9b0f7826c: Status 404 returned error can't find the container with id cfd99789acc2ec1f347c278306c048a2dba6861a32c6128b8f86c1d9b0f7826c Oct 30 06:24:17.392839 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.392807 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" podUID="045d509b-c8d2-4e2b-a687-01058f2097a4" containerName="payload-processing" containerID="cri-o://1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff" gracePeriod=25 Oct 30 06:24:17.552272 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:17.552230 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:24:17.664485 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.664456 5565 patch_prober.go:29] interesting pod/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:17.664651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.664512 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:17.666856 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.666837 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 06:24:17.794294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.794261 5565 generic.go:356] "Generic (PLEG): container finished" podID="045d509b-c8d2-4e2b-a687-01058f2097a4" containerID="1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff" exitCode=0 Oct 30 06:24:17.794471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.794312 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" Oct 30 06:24:17.794471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.794342 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" event={"ID":"045d509b-c8d2-4e2b-a687-01058f2097a4","Type":"ContainerDied","Data":"1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff"} Oct 30 06:24:17.794471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.794400 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl" event={"ID":"045d509b-c8d2-4e2b-a687-01058f2097a4","Type":"ContainerDied","Data":"e8fd169139269ad1d13c5bea091734360272ad3279cf05abd287a6f4dc88419f"} Oct 30 06:24:17.794471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.794425 5565 scope.go:122] "RemoveContainer" containerID="1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff" Oct 30 06:24:17.796181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.796158 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" event={"ID":"7a13018c-df26-401e-adaa-0a4bdd97e52a","Type":"ContainerStarted","Data":"b4f824e897fb46a5686500b4a726fcd9225de648a6c75d06ed2760e185f7ac33"} Oct 30 06:24:17.796289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.796211 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" event={"ID":"7a13018c-df26-401e-adaa-0a4bdd97e52a","Type":"ContainerStarted","Data":"cfd99789acc2ec1f347c278306c048a2dba6861a32c6128b8f86c1d9b0f7826c"} Oct 30 06:24:17.804698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.804681 5565 scope.go:122] "RemoveContainer" containerID="1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff" Oct 30 06:24:17.804968 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:17.804947 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff\": container with ID starting with 1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff not found: ID does not exist" containerID="1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff" Oct 30 06:24:17.805027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.804981 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff"} err="failed to get container status \"1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff\": rpc error: code = NotFound desc = could not find container \"1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff\": container with ID starting with 1f0e0140dcb75664e06158cfeab87d6ba0af82719a113c1d9f4aea09200598ff not found: ID does not exist" Oct 30 06:24:17.808311 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.808294 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw\" (UniqueName: \"kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw\") pod \"045d509b-c8d2-4e2b-a687-01058f2097a4\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " Oct 30 06:24:17.808389 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.808346 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume\") pod \"045d509b-c8d2-4e2b-a687-01058f2097a4\" (UID: \"045d509b-c8d2-4e2b-a687-01058f2097a4\") " Oct 30 06:24:17.808711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.808689 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume" pod "045d509b-c8d2-4e2b-a687-01058f2097a4" (UID: "045d509b-c8d2-4e2b-a687-01058f2097a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:17.808797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.808783 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/045d509b-c8d2-4e2b-a687-01058f2097a4-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:17.810393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.810371 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw" pod "045d509b-c8d2-4e2b-a687-01058f2097a4" (UID: "045d509b-c8d2-4e2b-a687-01058f2097a4"). InnerVolumeSpecName "kube-api-access-zkfxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:17.860186 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.860123 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" podStartSLOduration=1.860106593 podStartE2EDuration="1.860106593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:17.858543236 +0000 UTC m=+3164.434230292" watchObservedRunningTime="2026-10-30 06:24:17.860106593 +0000 UTC m=+3164.435793650" Oct 30 06:24:17.910229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:17.910173 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfxw\" (UniqueName: \"kubernetes.io/projected/045d509b-c8d2-4e2b-a687-01058f2097a4-kube-api-access-zkfxw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.158162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.158134 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl"] Oct 30 06:24:18.171157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.171131 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-64123f-55b88846fd-gj4rl"] Oct 30 06:24:18.392078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.392055 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:24:18.515740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.515710 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.515910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.515747 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.515910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.515793 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.515910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.515855 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.515910 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.515910 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.516208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.515972 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.516208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516022 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.516208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516063 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.516208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516120 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data\") pod \"efb6f096-bce0-47bc-bd26-6049d733d934\" (UID: \"efb6f096-bce0-47bc-bd26-6049d733d934\") " Oct 30 06:24:18.516208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516168 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.516688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516566 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.516802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516752 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.517117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516896 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.517117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.516936 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.517117 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.517077 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:18.518355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.518325 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:18.518471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.518446 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "kube-api-access-k77hd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:18.518759 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.518740 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:18.518832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.518780 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token" pod "efb6f096-bce0-47bc-bd26-6049d733d934" (UID: "efb6f096-bce0-47bc-bd26-6049d733d934"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:18.613284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.613249 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s"] Oct 30 06:24:18.617396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617370 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617393 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617403 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k77hd\" (UniqueName: \"kubernetes.io/projected/efb6f096-bce0-47bc-bd26-6049d733d934-kube-api-access-k77hd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617413 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617421 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617429 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/efb6f096-bce0-47bc-bd26-6049d733d934-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617437 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/efb6f096-bce0-47bc-bd26-6049d733d934-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.617566 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.617446 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/efb6f096-bce0-47bc-bd26-6049d733d934-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:18.804469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.804394 5565 generic.go:356] "Generic (PLEG): container finished" podID="7a13018c-df26-401e-adaa-0a4bdd97e52a" containerID="b4f824e897fb46a5686500b4a726fcd9225de648a6c75d06ed2760e185f7ac33" exitCode=0 Oct 30 06:24:18.804608 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.804482 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" event={"ID":"7a13018c-df26-401e-adaa-0a4bdd97e52a","Type":"ContainerDied","Data":"b4f824e897fb46a5686500b4a726fcd9225de648a6c75d06ed2760e185f7ac33"} Oct 30 06:24:18.805851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.805828 5565 generic.go:356] "Generic (PLEG): container finished" podID="efb6f096-bce0-47bc-bd26-6049d733d934" containerID="0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507" exitCode=0 Oct 30 06:24:18.805939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.805869 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" Oct 30 06:24:18.805939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.805892 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" event={"ID":"efb6f096-bce0-47bc-bd26-6049d733d934","Type":"ContainerDied","Data":"0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507"} Oct 30 06:24:18.805939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.805930 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4" event={"ID":"efb6f096-bce0-47bc-bd26-6049d733d934","Type":"ContainerDied","Data":"d7b9cb916408777fd8b3e6671d18670879a2173f568907c726d4714bb2e12c45"} Oct 30 06:24:18.806055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.805953 5565 scope.go:122] "RemoveContainer" containerID="0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507" Oct 30 06:24:18.819307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.819287 5565 scope.go:122] "RemoveContainer" containerID="0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507" Oct 30 06:24:18.819617 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:18.819599 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507\": container with ID starting with 0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507 not found: ID does not exist" containerID="0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507" Oct 30 06:24:18.819700 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.819623 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507"} err="failed to get container status \"0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507\": rpc error: code = NotFound desc = could not find container \"0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507\": container with ID starting with 0e32e19ec54647b2c82eceb53c7ff26bd36a5e0a9480e7bb42d51084e8db0507 not found: ID does not exist" Oct 30 06:24:18.895371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.894012 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4"] Oct 30 06:24:18.898289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:18.898260 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-2f1e4c-openshift-default-5668479585-czlh4"] Oct 30 06:24:19.566789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:19.566755 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="045d509b-c8d2-4e2b-a687-01058f2097a4" path="/var/lib/kubelet/pods/045d509b-c8d2-4e2b-a687-01058f2097a4/volumes" Oct 30 06:24:19.567214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:19.567201 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" path="/var/lib/kubelet/pods/efb6f096-bce0-47bc-bd26-6049d733d934/volumes" Oct 30 06:24:19.944384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:19.944360 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:20.029835 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.029809 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca\") pod \"7a13018c-df26-401e-adaa-0a4bdd97e52a\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " Oct 30 06:24:20.029974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.029894 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca\") pod \"7a13018c-df26-401e-adaa-0a4bdd97e52a\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " Oct 30 06:24:20.029974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.029947 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw\" (UniqueName: \"kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw\") pod \"7a13018c-df26-401e-adaa-0a4bdd97e52a\" (UID: \"7a13018c-df26-401e-adaa-0a4bdd97e52a\") " Oct 30 06:24:20.030160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.030140 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca" pod "7a13018c-df26-401e-adaa-0a4bdd97e52a" (UID: "7a13018c-df26-401e-adaa-0a4bdd97e52a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:20.030244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.030231 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:20.032079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.032058 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca" pod "7a13018c-df26-401e-adaa-0a4bdd97e52a" (UID: "7a13018c-df26-401e-adaa-0a4bdd97e52a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:20.032079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.032065 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw" pod "7a13018c-df26-401e-adaa-0a4bdd97e52a" (UID: "7a13018c-df26-401e-adaa-0a4bdd97e52a"). InnerVolumeSpecName "kube-api-access-ff7pw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:20.130926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.130873 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ff7pw\" (UniqueName: \"kubernetes.io/projected/7a13018c-df26-401e-adaa-0a4bdd97e52a-kube-api-access-ff7pw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:20.130926 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.130895 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7a13018c-df26-401e-adaa-0a4bdd97e52a-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:20.819974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.819945 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" Oct 30 06:24:20.819974 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.819966 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45" event={"ID":"7a13018c-df26-401e-adaa-0a4bdd97e52a","Type":"ContainerDied","Data":"cfd99789acc2ec1f347c278306c048a2dba6861a32c6128b8f86c1d9b0f7826c"} Oct 30 06:24:20.820188 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:20.820007 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfd99789acc2ec1f347c278306c048a2dba6861a32c6128b8f86c1d9b0f7826c" Oct 30 06:24:21.620622 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:21.620586 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:21.620622 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:21.620605 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:21.864591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:21.864554 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" containerID="cri-o://3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a" gracePeriod=45 Oct 30 06:24:21.877689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:21.877623 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" containerID="cri-o://848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22" gracePeriod=45 Oct 30 06:24:22.142379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.142320 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 06:24:22.162757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.162737 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 06:24:22.250663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250629 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg\" (UniqueName: \"kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg\") pod \"1c38aaef-b590-4231-bb2f-d311af3e8002\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " Oct 30 06:24:22.250838 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250713 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs\" (UniqueName: \"kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs\") pod \"9982d791-7470-4816-9b2b-7229801b63d7\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " Oct 30 06:24:22.250838 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250755 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache\") pod \"9982d791-7470-4816-9b2b-7229801b63d7\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " Oct 30 06:24:22.250838 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250820 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir\") pod \"1c38aaef-b590-4231-bb2f-d311af3e8002\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " Oct 30 06:24:22.250964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250877 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs\") pod \"1c38aaef-b590-4231-bb2f-d311af3e8002\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " Oct 30 06:24:22.250964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250931 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm\") pod \"1c38aaef-b590-4231-bb2f-d311af3e8002\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " Oct 30 06:24:22.251054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.250976 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home\") pod \"9982d791-7470-4816-9b2b-7229801b63d7\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " Oct 30 06:24:22.251054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251012 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6\" (UniqueName: \"kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6\") pod \"9982d791-7470-4816-9b2b-7229801b63d7\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " Oct 30 06:24:22.251156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251060 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home\") pod \"1c38aaef-b590-4231-bb2f-d311af3e8002\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " Oct 30 06:24:22.251156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251104 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir\") pod \"9982d791-7470-4816-9b2b-7229801b63d7\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " Oct 30 06:24:22.251156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251135 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm\") pod \"9982d791-7470-4816-9b2b-7229801b63d7\" (UID: \"9982d791-7470-4816-9b2b-7229801b63d7\") " Oct 30 06:24:22.251304 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251167 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache\") pod \"1c38aaef-b590-4231-bb2f-d311af3e8002\" (UID: \"1c38aaef-b590-4231-bb2f-d311af3e8002\") " Oct 30 06:24:22.252135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251599 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir" pod "9982d791-7470-4816-9b2b-7229801b63d7" (UID: "9982d791-7470-4816-9b2b-7229801b63d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.252135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251825 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir" pod "1c38aaef-b590-4231-bb2f-d311af3e8002" (UID: "1c38aaef-b590-4231-bb2f-d311af3e8002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.252135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.251837 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home" pod "1c38aaef-b590-4231-bb2f-d311af3e8002" (UID: "1c38aaef-b590-4231-bb2f-d311af3e8002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.252135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.252125 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache" pod "9982d791-7470-4816-9b2b-7229801b63d7" (UID: "9982d791-7470-4816-9b2b-7229801b63d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.252476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.252304 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home" pod "9982d791-7470-4816-9b2b-7229801b63d7" (UID: "9982d791-7470-4816-9b2b-7229801b63d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.252586 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.252560 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache" pod "1c38aaef-b590-4231-bb2f-d311af3e8002" (UID: "1c38aaef-b590-4231-bb2f-d311af3e8002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.254796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.254769 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg" pod "1c38aaef-b590-4231-bb2f-d311af3e8002" (UID: "1c38aaef-b590-4231-bb2f-d311af3e8002"). InnerVolumeSpecName "kube-api-access-q5nwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:22.254925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.254803 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm" pod "9982d791-7470-4816-9b2b-7229801b63d7" (UID: "9982d791-7470-4816-9b2b-7229801b63d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.255148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.255099 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm" pod "1c38aaef-b590-4231-bb2f-d311af3e8002" (UID: "1c38aaef-b590-4231-bb2f-d311af3e8002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:22.255224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.255200 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6" pod "9982d791-7470-4816-9b2b-7229801b63d7" (UID: "9982d791-7470-4816-9b2b-7229801b63d7"). InnerVolumeSpecName "kube-api-access-fddb6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:22.255284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.255201 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs" pod "9982d791-7470-4816-9b2b-7229801b63d7" (UID: "9982d791-7470-4816-9b2b-7229801b63d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:22.256193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.256172 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs" pod "1c38aaef-b590-4231-bb2f-d311af3e8002" (UID: "1c38aaef-b590-4231-bb2f-d311af3e8002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:22.353140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353102 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353132 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353142 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fddb6\" (UniqueName: \"kubernetes.io/projected/9982d791-7470-4816-9b2b-7229801b63d7-kube-api-access-fddb6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353153 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353183 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353192 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353200 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353208 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5nwg\" (UniqueName: \"kubernetes.io/projected/1c38aaef-b590-4231-bb2f-d311af3e8002-kube-api-access-q5nwg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353216 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9982d791-7470-4816-9b2b-7229801b63d7-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353224 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9982d791-7470-4816-9b2b-7229801b63d7-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353231 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c38aaef-b590-4231-bb2f-d311af3e8002-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.353380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.353239 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c38aaef-b590-4231-bb2f-d311af3e8002-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:22.832256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.832220 5565 generic.go:356] "Generic (PLEG): container finished" podID="9982d791-7470-4816-9b2b-7229801b63d7" containerID="3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a" exitCode=0 Oct 30 06:24:22.832747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.832270 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" Oct 30 06:24:22.832747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.832307 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" event={"ID":"9982d791-7470-4816-9b2b-7229801b63d7","Type":"ContainerDied","Data":"3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a"} Oct 30 06:24:22.832747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.832372 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp" event={"ID":"9982d791-7470-4816-9b2b-7229801b63d7","Type":"ContainerDied","Data":"643d47e5bbde6db2d57004f33fc169f104f51f1bf1915bb76d2b9a3759a78ae6"} Oct 30 06:24:22.832747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.832398 5565 scope.go:122] "RemoveContainer" containerID="3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a" Oct 30 06:24:22.833913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.833892 5565 generic.go:356] "Generic (PLEG): container finished" podID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerID="848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22" exitCode=0 Oct 30 06:24:22.834023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.833919 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" event={"ID":"1c38aaef-b590-4231-bb2f-d311af3e8002","Type":"ContainerDied","Data":"848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22"} Oct 30 06:24:22.834023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.833939 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" Oct 30 06:24:22.834023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.833961 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245" event={"ID":"1c38aaef-b590-4231-bb2f-d311af3e8002","Type":"ContainerDied","Data":"9305c0c53d542ca7550977ff8fdeaad5f374d6528780bc67a038ca2a0e611876"} Oct 30 06:24:22.843093 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.843078 5565 scope.go:122] "RemoveContainer" containerID="3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a" Oct 30 06:24:22.843324 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:22.843303 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a\": container with ID starting with 3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a not found: ID does not exist" containerID="3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a" Oct 30 06:24:22.843386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.843331 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a"} err="failed to get container status \"3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a\": rpc error: code = NotFound desc = could not find container \"3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a\": container with ID starting with 3e3c5cc834070453c3b50f90471c2edf23f3c46c9c3a611b146dd22d156d700a not found: ID does not exist" Oct 30 06:24:22.843386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.843350 5565 scope.go:122] "RemoveContainer" containerID="848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22" Oct 30 06:24:22.852259 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.852243 5565 scope.go:122] "RemoveContainer" containerID="848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22" Oct 30 06:24:22.852597 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:22.852516 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22\": container with ID starting with 848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22 not found: ID does not exist" containerID="848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22" Oct 30 06:24:22.852650 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.852608 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22"} err="failed to get container status \"848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22\": rpc error: code = NotFound desc = could not find container \"848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22\": container with ID starting with 848fdca935143eff7018906c439ea6352dbb5737698697987e3658b2f0e2bb22 not found: ID does not exist" Oct 30 06:24:22.888371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.888337 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245"] Oct 30 06:24:22.902239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.902215 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/simulator-w6-64123f-kserve-579dfb4489-zr245"] Oct 30 06:24:22.929880 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.929861 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp"] Oct 30 06:24:22.936270 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:22.936248 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-64123f/distinct-w6-64123f-kserve-745f554c55-lqqxp"] Oct 30 06:24:23.565471 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:23.565439 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" path="/var/lib/kubelet/pods/1c38aaef-b590-4231-bb2f-d311af3e8002/volumes" Oct 30 06:24:23.565948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:23.565932 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9982d791-7470-4816-9b2b-7229801b63d7" path="/var/lib/kubelet/pods/9982d791-7470-4816-9b2b-7229801b63d7/volumes" Oct 30 06:24:24.183362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.183327 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd"] Oct 30 06:24:24.594636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.594606 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl"] Oct 30 06:24:24.595202 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595177 5565 cpu_manager.go:414] "containerMap: removing container" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" Oct 30 06:24:24.595202 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595199 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595240 5565 cpu_manager.go:414] "containerMap: removing container" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595249 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595279 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7a13018c-df26-401e-adaa-0a4bdd97e52a" containerName="revoke-keys" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595287 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a13018c-df26-401e-adaa-0a4bdd97e52a" containerName="revoke-keys" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595301 5565 cpu_manager.go:414] "containerMap: removing container" podUID="045d509b-c8d2-4e2b-a687-01058f2097a4" containerName="payload-processing" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595308 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="045d509b-c8d2-4e2b-a687-01058f2097a4" containerName="payload-processing" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595317 5565 cpu_manager.go:414] "containerMap: removing container" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595325 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595422 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a13018c-df26-401e-adaa-0a4bdd97e52a" containerName="revoke-keys" Oct 30 06:24:24.595427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595438 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="045d509b-c8d2-4e2b-a687-01058f2097a4" containerName="payload-processing" Oct 30 06:24:24.595982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595447 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="9982d791-7470-4816-9b2b-7229801b63d7" containerName="main" Oct 30 06:24:24.595982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595458 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c38aaef-b590-4231-bb2f-d311af3e8002" containerName="main" Oct 30 06:24:24.595982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.595471 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="efb6f096-bce0-47bc-bd26-6049d733d934" containerName="istio-proxy" Oct 30 06:24:24.600379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.600357 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.619692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.619626 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl"] Oct 30 06:24:24.620826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.620802 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:24:24.620826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.620810 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:24:24.680015 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.679978 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qg4\" (UniqueName: \"kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.680198 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.680063 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.680292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.680263 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.695883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.695853 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:24:24.696088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.695967 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:24:24.781455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.781424 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.781651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.781521 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.781651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.781574 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qg4\" (UniqueName: \"kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.782241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.782221 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.783873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.783847 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.806807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.806784 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qg4\" (UniqueName: \"kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4\") pod \"maas-api-revoke-sub-af64ac97-w97hl\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:24.913751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:24.913694 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:25.214925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:25.214903 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl"] Oct 30 06:24:25.217090 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:24:25.217054 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2bf6f0d_9bc7_4d7b_a19c_5d1f06e62f99.slice/crio-a91bdb5b9d4081197021d61b1b30992978f2c0ce714d5c6043af69d10a44efee WatchSource:0}: Error finding container a91bdb5b9d4081197021d61b1b30992978f2c0ce714d5c6043af69d10a44efee: Status 404 returned error can't find the container with id a91bdb5b9d4081197021d61b1b30992978f2c0ce714d5c6043af69d10a44efee Oct 30 06:24:25.855016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:25.854982 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" event={"ID":"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99","Type":"ContainerStarted","Data":"62b27f1dea0838713ca4e65445642fb1d5f3495de7701532cda04aae9556c10e"} Oct 30 06:24:25.855229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:25.855040 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" event={"ID":"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99","Type":"ContainerStarted","Data":"a91bdb5b9d4081197021d61b1b30992978f2c0ce714d5c6043af69d10a44efee"} Oct 30 06:24:25.899354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:25.899293 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" podStartSLOduration=1.899280052 podStartE2EDuration="1.899280052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:24:25.89895415 +0000 UTC m=+3172.474641203" watchObservedRunningTime="2026-10-30 06:24:25.899280052 +0000 UTC m=+3172.474967107" Oct 30 06:24:26.352598 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.352577 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 06:24:26.355417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.355396 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 06:24:26.400663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.400605 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume\") pod \"cc496ad6-4aea-4eb6-acc0-001f579881bf\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " Oct 30 06:24:26.400788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.400702 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume\") pod \"6e26684e-e7b1-4118-b69e-c5375a0586b3\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " Oct 30 06:24:26.400788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.400759 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj\" (UniqueName: \"kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj\") pod \"6e26684e-e7b1-4118-b69e-c5375a0586b3\" (UID: \"6e26684e-e7b1-4118-b69e-c5375a0586b3\") " Oct 30 06:24:26.400890 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.400794 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb\" (UniqueName: \"kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb\") pod \"cc496ad6-4aea-4eb6-acc0-001f579881bf\" (UID: \"cc496ad6-4aea-4eb6-acc0-001f579881bf\") " Oct 30 06:24:26.400996 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.400973 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume" pod "cc496ad6-4aea-4eb6-acc0-001f579881bf" (UID: "cc496ad6-4aea-4eb6-acc0-001f579881bf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:26.401138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.401110 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume" pod "6e26684e-e7b1-4118-b69e-c5375a0586b3" (UID: "6e26684e-e7b1-4118-b69e-c5375a0586b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:26.401218 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.401186 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc496ad6-4aea-4eb6-acc0-001f579881bf-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:26.403284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.403257 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj" pod "6e26684e-e7b1-4118-b69e-c5375a0586b3" (UID: "6e26684e-e7b1-4118-b69e-c5375a0586b3"). InnerVolumeSpecName "kube-api-access-stnqj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:26.403428 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.403404 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb" pod "cc496ad6-4aea-4eb6-acc0-001f579881bf" (UID: "cc496ad6-4aea-4eb6-acc0-001f579881bf"). InnerVolumeSpecName "kube-api-access-s7bbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:26.502636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.502597 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6e26684e-e7b1-4118-b69e-c5375a0586b3-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:26.502636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.502631 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-stnqj\" (UniqueName: \"kubernetes.io/projected/6e26684e-e7b1-4118-b69e-c5375a0586b3-kube-api-access-stnqj\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:26.502845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.502645 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s7bbb\" (UniqueName: \"kubernetes.io/projected/cc496ad6-4aea-4eb6-acc0-001f579881bf-kube-api-access-s7bbb\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:26.862661 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.862626 5565 generic.go:356] "Generic (PLEG): container finished" podID="b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" containerID="62b27f1dea0838713ca4e65445642fb1d5f3495de7701532cda04aae9556c10e" exitCode=0 Oct 30 06:24:26.862826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.862711 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" event={"ID":"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99","Type":"ContainerDied","Data":"62b27f1dea0838713ca4e65445642fb1d5f3495de7701532cda04aae9556c10e"} Oct 30 06:24:26.864081 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.864059 5565 generic.go:356] "Generic (PLEG): container finished" podID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerID="9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9" exitCode=0 Oct 30 06:24:26.864229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.864118 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" Oct 30 06:24:26.864229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.864121 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" event={"ID":"6e26684e-e7b1-4118-b69e-c5375a0586b3","Type":"ContainerDied","Data":"9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9"} Oct 30 06:24:26.864353 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.864242 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd" event={"ID":"6e26684e-e7b1-4118-b69e-c5375a0586b3","Type":"ContainerDied","Data":"3f04994b9fcee97b14e8c9c941abeb5b4e3a5a658f6fcb3db77feb263b4f9331"} Oct 30 06:24:26.864353 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.864273 5565 scope.go:122] "RemoveContainer" containerID="9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9" Oct 30 06:24:26.865470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.865448 5565 generic.go:356] "Generic (PLEG): container finished" podID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerID="ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13" exitCode=0 Oct 30 06:24:26.865607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.865509 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" Oct 30 06:24:26.865607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.865559 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" event={"ID":"cc496ad6-4aea-4eb6-acc0-001f579881bf","Type":"ContainerDied","Data":"ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13"} Oct 30 06:24:26.865607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.865596 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds" event={"ID":"cc496ad6-4aea-4eb6-acc0-001f579881bf","Type":"ContainerDied","Data":"5e6ffbe1eaa4444cbd8c0b91d6a0002f4a41cf63469f06d12042db03bf5d990c"} Oct 30 06:24:26.875224 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.875204 5565 scope.go:122] "RemoveContainer" containerID="9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9" Oct 30 06:24:26.875793 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:26.875767 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9\": container with ID starting with 9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9 not found: ID does not exist" containerID="9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9" Oct 30 06:24:26.875913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.875803 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9"} err="failed to get container status \"9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9\": rpc error: code = NotFound desc = could not find container \"9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9\": container with ID starting with 9a02ab642f8fbbf2f25dea4faf4db06e4fcb67dfd1ed057f1aad42d317794dd9 not found: ID does not exist" Oct 30 06:24:26.875913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.875827 5565 scope.go:122] "RemoveContainer" containerID="ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13" Oct 30 06:24:26.884604 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.884589 5565 scope.go:122] "RemoveContainer" containerID="ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13" Oct 30 06:24:26.884874 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:26.884852 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13\": container with ID starting with ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13 not found: ID does not exist" containerID="ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13" Oct 30 06:24:26.884949 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.884878 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13"} err="failed to get container status \"ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13\": rpc error: code = NotFound desc = could not find container \"ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13\": container with ID starting with ee9c809761b39f0beaa954394147657c2e925c088aab714150a1c04643b55e13 not found: ID does not exist" Oct 30 06:24:26.946102 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.946074 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd"] Oct 30 06:24:26.950947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.950924 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-1d8c2f-6b5857df48-thpfd"] Oct 30 06:24:26.987913 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.987885 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds"] Oct 30 06:24:26.992647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:26.992622 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-1d8c2f-6cfbd7dc59-wprds"] Oct 30 06:24:27.566311 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:27.566276 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" path="/var/lib/kubelet/pods/6e26684e-e7b1-4118-b69e-c5375a0586b3/volumes" Oct 30 06:24:27.566703 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:27.566661 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" path="/var/lib/kubelet/pods/cc496ad6-4aea-4eb6-acc0-001f579881bf/volumes" Oct 30 06:24:28.005095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.005070 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:28.117366 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.117335 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca\") pod \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " Oct 30 06:24:28.117498 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.117460 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca\") pod \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " Oct 30 06:24:28.117564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.117500 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4\" (UniqueName: \"kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4\") pod \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\" (UID: \"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99\") " Oct 30 06:24:28.117869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.117841 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca" pod "b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" (UID: "b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:28.119688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.119662 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca" pod "b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" (UID: "b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:28.119811 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.119709 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4" pod "b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" (UID: "b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99"). InnerVolumeSpecName "kube-api-access-r2qg4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:28.218245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.218215 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.218245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.218237 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2qg4\" (UniqueName: \"kubernetes.io/projected/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-kube-api-access-r2qg4\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.218245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.218248 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:28.880325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.880288 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" Oct 30 06:24:28.880757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.880331 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl" event={"ID":"b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99","Type":"ContainerDied","Data":"a91bdb5b9d4081197021d61b1b30992978f2c0ce714d5c6043af69d10a44efee"} Oct 30 06:24:28.880757 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:28.880381 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a91bdb5b9d4081197021d61b1b30992978f2c0ce714d5c6043af69d10a44efee" Oct 30 06:24:29.184243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:29.184163 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerName="payload-processing" containerID="cri-o://051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e" gracePeriod=25 Oct 30 06:24:32.994266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:32.994225 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:24:33.633144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.633097 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" containerID="cri-o://15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf" gracePeriod=45 Oct 30 06:24:33.891578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.891510 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 06:24:33.911520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.911488 5565 generic.go:356] "Generic (PLEG): container finished" podID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerID="15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf" exitCode=0 Oct 30 06:24:33.911679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.911549 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" Oct 30 06:24:33.911679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.911581 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" event={"ID":"d4b3adc8-3939-4a61-b5c1-8c69f82c3361","Type":"ContainerDied","Data":"15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf"} Oct 30 06:24:33.911679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.911636 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s" event={"ID":"d4b3adc8-3939-4a61-b5c1-8c69f82c3361","Type":"ContainerDied","Data":"f615b631b9f4037da07876e0ffc717353f9a0afc5ce73f80b1db802c401ffcff"} Oct 30 06:24:33.911679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.911655 5565 scope.go:122] "RemoveContainer" containerID="15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf" Oct 30 06:24:33.921405 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.921382 5565 scope.go:122] "RemoveContainer" containerID="15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf" Oct 30 06:24:33.921720 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:33.921696 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf\": container with ID starting with 15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf not found: ID does not exist" containerID="15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf" Oct 30 06:24:33.921826 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.921728 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf"} err="failed to get container status \"15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf\": rpc error: code = NotFound desc = could not find container \"15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf\": container with ID starting with 15f23d072d05a60ae1810229e2de8c1d45a51d89c95bb1872b475555b7d06dcf not found: ID does not exist" Oct 30 06:24:33.976510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976479 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache\") pod \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " Oct 30 06:24:33.976701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976523 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home\") pod \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " Oct 30 06:24:33.976701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976657 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs\") pod \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " Oct 30 06:24:33.976701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976683 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm\") pod \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " Oct 30 06:24:33.976888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976717 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt\" (UniqueName: \"kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt\") pod \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " Oct 30 06:24:33.976888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976752 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir\") pod \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\" (UID: \"d4b3adc8-3939-4a61-b5c1-8c69f82c3361\") " Oct 30 06:24:33.976888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976815 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache" pod "d4b3adc8-3939-4a61-b5c1-8c69f82c3361" (UID: "d4b3adc8-3939-4a61-b5c1-8c69f82c3361"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.977044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.976904 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home" pod "d4b3adc8-3939-4a61-b5c1-8c69f82c3361" (UID: "d4b3adc8-3939-4a61-b5c1-8c69f82c3361"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.977162 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.977131 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir" pod "d4b3adc8-3939-4a61-b5c1-8c69f82c3361" (UID: "d4b3adc8-3939-4a61-b5c1-8c69f82c3361"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.977296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.977204 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:33.977296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.977228 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:33.980160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.979650 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm" pod "d4b3adc8-3939-4a61-b5c1-8c69f82c3361" (UID: "d4b3adc8-3939-4a61-b5c1-8c69f82c3361"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:33.980160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.979680 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt" pod "d4b3adc8-3939-4a61-b5c1-8c69f82c3361" (UID: "d4b3adc8-3939-4a61-b5c1-8c69f82c3361"). InnerVolumeSpecName "kube-api-access-mj4jt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:33.980160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:33.980125 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs" pod "d4b3adc8-3939-4a61-b5c1-8c69f82c3361" (UID: "d4b3adc8-3939-4a61-b5c1-8c69f82c3361"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:24:34.078172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.078139 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.078172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.078169 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.078172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.078179 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mj4jt\" (UniqueName: \"kubernetes.io/projected/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-kube-api-access-mj4jt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.078653 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.078188 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d4b3adc8-3939-4a61-b5c1-8c69f82c3361-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:34.261888 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.261859 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s"] Oct 30 06:24:34.265845 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.265819 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-be78f2/distinct-w5-be78f2-kserve-7cb54887c8-5g64s"] Oct 30 06:24:34.638513 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:34.638479 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:35.565698 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:35.565666 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" path="/var/lib/kubelet/pods/d4b3adc8-3939-4a61-b5c1-8c69f82c3361/volumes" Oct 30 06:24:36.783033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:36.782998 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh"] Oct 30 06:24:36.783465 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:36.783237 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" containerID="cri-o://4c1d8d6b4a577d957208d4b592b408a2924f6243b046f824f5a6be7e16fa00e4" gracePeriod=30 Oct 30 06:24:39.559324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:39.559292 5565 scope.go:122] "RemoveContainer" containerID="9206785fefaef55656483806518b961fc4c9863e9e840a11a6492773fa827d41" Oct 30 06:24:39.568398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:39.568376 5565 scope.go:122] "RemoveContainer" containerID="71e999277be4ee6f165e05d084d93848c2c36d10a02980a8119515ed97bb9c33" Oct 30 06:24:39.576808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:39.576785 5565 scope.go:122] "RemoveContainer" containerID="3b148ffb0ff11e454a59bb28244c43e5422e23402870bf96a509ac1e2cb7a8b0" Oct 30 06:24:40.199167 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:40.199134 5565 patch_prober.go:29] interesting pod/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 06:24:40.199333 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:40.199205 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 06:24:40.700951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:40.700915 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:24:41.965472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:41.965436 5565 generic.go:356] "Generic (PLEG): container finished" podID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerID="4c1d8d6b4a577d957208d4b592b408a2924f6243b046f824f5a6be7e16fa00e4" exitCode=0 Oct 30 06:24:41.965851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:41.965473 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" event={"ID":"bd67151c-42b7-4c88-8f69-a90be47788c4","Type":"ContainerDied","Data":"4c1d8d6b4a577d957208d4b592b408a2924f6243b046f824f5a6be7e16fa00e4"} Oct 30 06:24:42.036808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.036785 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 06:24:42.161139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161049 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161106 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161139 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161136 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161156 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161231 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161255 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161274 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161333 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161334 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161374 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token\") pod \"bd67151c-42b7-4c88-8f69-a90be47788c4\" (UID: \"bd67151c-42b7-4c88-8f69-a90be47788c4\") " Oct 30 06:24:42.161505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161447 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.161958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161561 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.161958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161578 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.161958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161775 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.161958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161794 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.161958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161809 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.161958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161822 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.162295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.161982 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:42.163618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.163589 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "kube-api-access-8w8lt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:42.163881 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.163855 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:42.163961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.163904 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:42.163961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.163918 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token" pod "bd67151c-42b7-4c88-8f69-a90be47788c4" (UID: "bd67151c-42b7-4c88-8f69-a90be47788c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:42.263256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.263227 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.263256 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.263254 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w8lt\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-kube-api-access-8w8lt\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.263408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.263265 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd67151c-42b7-4c88-8f69-a90be47788c4-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.263408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.263273 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.263408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.263281 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd67151c-42b7-4c88-8f69-a90be47788c4-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:42.972712 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.972682 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" Oct 30 06:24:42.973172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.972709 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh" event={"ID":"bd67151c-42b7-4c88-8f69-a90be47788c4","Type":"ContainerDied","Data":"44658fd93b065e5acd5843748d67f94d8482117995230d9a5d1cd589f502715d"} Oct 30 06:24:42.973172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:42.972781 5565 scope.go:122] "RemoveContainer" containerID="4c1d8d6b4a577d957208d4b592b408a2924f6243b046f824f5a6be7e16fa00e4" Oct 30 06:24:43.044088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:43.044055 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh"] Oct 30 06:24:43.057908 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:43.057882 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-64123f-openshift-default-5f4b678889-j5tjh"] Oct 30 06:24:43.566298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:43.566266 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" path="/var/lib/kubelet/pods/bd67151c-42b7-4c88-8f69-a90be47788c4/volumes" Oct 30 06:24:44.638026 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:44.637987 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:24:46.286284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:46.286241 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd"] Oct 30 06:24:46.286905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:46.286489 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" containerID="cri-o://59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185" gracePeriod=30 Oct 30 06:24:47.531362 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:47.531327 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:24:49.424777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.424755 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 06:24:49.524691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.524664 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume\") pod \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " Oct 30 06:24:49.524830 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.524730 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh\" (UniqueName: \"kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh\") pod \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\" (UID: \"c52be5c8-a2c5-4617-9d9a-09e61374cfdd\") " Oct 30 06:24:49.525046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.525023 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume" pod "c52be5c8-a2c5-4617-9d9a-09e61374cfdd" (UID: "c52be5c8-a2c5-4617-9d9a-09e61374cfdd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:49.525135 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.525113 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:49.526749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.526728 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh" pod "c52be5c8-a2c5-4617-9d9a-09e61374cfdd" (UID: "c52be5c8-a2c5-4617-9d9a-09e61374cfdd"). InnerVolumeSpecName "kube-api-access-2xpjh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:49.625627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:49.625603 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xpjh\" (UniqueName: \"kubernetes.io/projected/c52be5c8-a2c5-4617-9d9a-09e61374cfdd-kube-api-access-2xpjh\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:50.010210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.010181 5565 generic.go:356] "Generic (PLEG): container finished" podID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerID="051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e" exitCode=0 Oct 30 06:24:50.010367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.010218 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" event={"ID":"c52be5c8-a2c5-4617-9d9a-09e61374cfdd","Type":"ContainerDied","Data":"051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e"} Oct 30 06:24:50.010367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.010223 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" Oct 30 06:24:50.010367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.010257 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd" event={"ID":"c52be5c8-a2c5-4617-9d9a-09e61374cfdd","Type":"ContainerDied","Data":"13b4ab62551c9c1c196eae011ed6a2998cb08f47f76affd617c995ed8b33eaec"} Oct 30 06:24:50.010367 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.010278 5565 scope.go:122] "RemoveContainer" containerID="051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e" Oct 30 06:24:50.019476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.019458 5565 scope.go:122] "RemoveContainer" containerID="051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e" Oct 30 06:24:50.019752 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:50.019731 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e\": container with ID starting with 051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e not found: ID does not exist" containerID="051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e" Oct 30 06:24:50.019807 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.019761 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e"} err="failed to get container status \"051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e\": rpc error: code = NotFound desc = could not find container \"051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e\": container with ID starting with 051631f7ac92349c23c296fc511f3ea5cd529c90575e85b3145f12ab3ff59b4e not found: ID does not exist" Oct 30 06:24:50.067973 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.067949 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd"] Oct 30 06:24:50.081756 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:50.081731 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-be78f2-5f8fb6649b-vdgdd"] Oct 30 06:24:51.548800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.548765 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 06:24:51.567210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.567133 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" path="/var/lib/kubelet/pods/c52be5c8-a2c5-4617-9d9a-09e61374cfdd/volumes" Oct 30 06:24:51.645199 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645170 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645259 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645291 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645324 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645353 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645409 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645556 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645600 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645694 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket\") pod \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\" (UID: \"7185eafe-7e10-483b-a8c6-afd9ea1dbf56\") " Oct 30 06:24:51.645824 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645730 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:51.646097 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.645936 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:51.646200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.646183 5565 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-data\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.646260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.646205 5565 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.646357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.646334 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:51.646793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.646767 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:51.646946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.646924 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:24:51.647840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.647815 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:51.647950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.647869 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:24:51.647950 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.647891 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "kube-api-access-f2mv6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:24:51.648104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.648087 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo" pod "7185eafe-7e10-483b-a8c6-afd9ea1dbf56" (UID: "7185eafe-7e10-483b-a8c6-afd9ea1dbf56"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 06:24:51.747560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747517 5565 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-podinfo\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.747560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747559 5565 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istiod-ca-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.747707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747570 5565 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-credential-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.747707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747579 5565 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-workload-socket\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.747707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747589 5565 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-envoy\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.747707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747599 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:51.747707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:51.747607 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2mv6\" (UniqueName: \"kubernetes.io/projected/7185eafe-7e10-483b-a8c6-afd9ea1dbf56-kube-api-access-f2mv6\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:24:52.026141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.026111 5565 generic.go:356] "Generic (PLEG): container finished" podID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerID="59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185" exitCode=0 Oct 30 06:24:52.026299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.026161 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" Oct 30 06:24:52.026299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.026192 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" event={"ID":"7185eafe-7e10-483b-a8c6-afd9ea1dbf56","Type":"ContainerDied","Data":"59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185"} Oct 30 06:24:52.026299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.026248 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd" event={"ID":"7185eafe-7e10-483b-a8c6-afd9ea1dbf56","Type":"ContainerDied","Data":"15bcd6c2e3e92269608221b1efcc40b946071c186335c809e1c8a0b9ec99b0a0"} Oct 30 06:24:52.026299 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.026269 5565 scope.go:122] "RemoveContainer" containerID="59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185" Oct 30 06:24:52.036827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.036810 5565 scope.go:122] "RemoveContainer" containerID="bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149" Oct 30 06:24:52.045319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.045303 5565 scope.go:122] "RemoveContainer" containerID="59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185" Oct 30 06:24:52.045591 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:52.045566 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185\": container with ID starting with 59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185 not found: ID does not exist" containerID="59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185" Oct 30 06:24:52.045649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.045608 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185"} err="failed to get container status \"59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185\": rpc error: code = NotFound desc = could not find container \"59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185\": container with ID starting with 59303e4dc8162bf7ca31f112295f6c8907012fbff5e1ff7fd50c6b546a0a8185 not found: ID does not exist" Oct 30 06:24:52.045649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.045631 5565 scope.go:122] "RemoveContainer" containerID="bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149" Oct 30 06:24:52.045848 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:24:52.045828 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149\": container with ID starting with bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149 not found: ID does not exist" containerID="bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149" Oct 30 06:24:52.045917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.045857 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149"} err="failed to get container status \"bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149\": rpc error: code = NotFound desc = could not find container \"bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149\": container with ID starting with bba3f964e935c001f48e29e44f5b89052eaeb84584095c2caeab94ec8bc83149 not found: ID does not exist" Oct 30 06:24:52.084318 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.084278 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd"] Oct 30 06:24:52.097329 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:52.097283 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-be78f2-openshift-default-84b7466d55-4hsvd"] Oct 30 06:24:53.565253 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:53.565213 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" path="/var/lib/kubelet/pods/7185eafe-7e10-483b-a8c6-afd9ea1dbf56/volumes" Oct 30 06:24:58.028153 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:58.028117 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m"] Oct 30 06:24:58.042016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:58.041990 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a47c8008-6w59m"] Oct 30 06:24:59.565798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:24:59.565767 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52953f74-1fbf-4c44-be7c-018905c5a213" path="/var/lib/kubelet/pods/52953f74-1fbf-4c44-be7c-018905c5a213/volumes" Oct 30 06:25:17.531393 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:25:17.531357 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:25:25.596669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.596637 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd"] Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.596987 5565 cpu_manager.go:414] "containerMap: removing container" podUID="b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" containerName="revoke-keys" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.596998 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" containerName="revoke-keys" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597012 5565 cpu_manager.go:414] "containerMap: removing container" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerName="payload-pre-processing" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597017 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerName="payload-pre-processing" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597031 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerName="payload-processing" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597036 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerName="payload-processing" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597049 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerName="payload-processing" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597053 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerName="payload-processing" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597066 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597071 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" Oct 30 06:25:25.597075 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597081 5565 cpu_manager.go:414] "containerMap: removing container" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597085 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597096 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597100 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597109 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597113 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597180 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" containerName="revoke-keys" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597189 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc496ad6-4aea-4eb6-acc0-001f579881bf" containerName="payload-pre-processing" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597196 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c52be5c8-a2c5-4617-9d9a-09e61374cfdd" containerName="payload-processing" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597202 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e26684e-e7b1-4118-b69e-c5375a0586b3" containerName="payload-processing" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597210 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd67151c-42b7-4c88-8f69-a90be47788c4" containerName="istio-proxy" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597217 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b3adc8-3939-4a61-b5c1-8c69f82c3361" containerName="main" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597223 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" Oct 30 06:25:25.597455 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.597230 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7185eafe-7e10-483b-a8c6-afd9ea1dbf56" containerName="istio-proxy" Oct 30 06:25:25.600175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.600156 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.632419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.632391 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd"] Oct 30 06:25:25.725785 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.725754 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.725944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.725796 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.725944 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.725893 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.726078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.725992 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.726078 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.726023 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz7f7\" (UniqueName: \"kubernetes.io/projected/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-kube-api-access-gz7f7\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.726176 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.726086 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.726176 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.726105 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.726176 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.726126 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.726335 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.726169 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827298 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827375 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz7f7\" (UniqueName: \"kubernetes.io/projected/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-kube-api-access-gz7f7\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827434 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827453 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827475 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827671 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827494 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827660 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827722 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827752 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827798 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827870 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.827939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.827934 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.828104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.828091 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.828104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.828093 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.829871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.829850 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.830170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.830151 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.852289 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.852236 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.852385 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.852367 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz7f7\" (UniqueName: \"kubernetes.io/projected/1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20-kube-api-access-gz7f7\") pod \"maas-default-gateway-openshift-default-84587756dc-bbbvd\" (UID: \"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:25.914276 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:25.914255 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:26.234138 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:26.234113 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd"] Oct 30 06:25:26.236231 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:25:26.236197 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d63c4f9_cde5_4b64_a152_9e8a6cb0ee20.slice/crio-85a010013747d308e41709a52f1767323c18599945a2a90d30df8b91f6785698 WatchSource:0}: Error finding container 85a010013747d308e41709a52f1767323c18599945a2a90d30df8b91f6785698: Status 404 returned error can't find the container with id 85a010013747d308e41709a52f1767323c18599945a2a90d30df8b91f6785698 Oct 30 06:25:26.238272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:26.238240 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:25:26.238372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:26.238301 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:25:26.238372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:26.238328 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:25:27.217142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:27.217108 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" event={"ID":"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20","Type":"ContainerStarted","Data":"69b6384904fdc386b531e8c1ca3ae5cd1ca99818b871af9eae3340f4ded3a1e3"} Oct 30 06:25:27.217583 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:27.217161 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" event={"ID":"1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20","Type":"ContainerStarted","Data":"85a010013747d308e41709a52f1767323c18599945a2a90d30df8b91f6785698"} Oct 30 06:25:27.293714 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:27.293658 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podStartSLOduration=2.293644636 podStartE2EDuration="2.293644636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:25:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:25:27.2797521 +0000 UTC m=+3233.855439165" watchObservedRunningTime="2026-10-30 06:25:27.293644636 +0000 UTC m=+3233.869331690" Oct 30 06:25:27.914821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:27.914787 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:27.916178 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:27.916151 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:25:27.916325 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:27.916211 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 30 06:25:28.915079 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:28.915033 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:25:28.915602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:28.915097 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 30 06:25:29.915609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:29.915572 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:25:29.916017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:29.915626 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 30 06:25:30.915469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:30.915437 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:25:30.915668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:30.915501 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 30 06:25:31.915181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:31.915142 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:25:31.915398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:31.915202 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 30 06:25:32.915574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:32.915508 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" start-of-body= Oct 30 06:25:32.915966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:32.915598 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": dial tcp 10.134.1.53:15021: connect: connection refused" Oct 30 06:25:34.654372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:34.654334 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:34.654925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:34.654748 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:34.655663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:34.655641 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" Oct 30 06:25:39.683574 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:39.683518 5565 scope.go:122] "RemoveContainer" containerID="684308037d5d7aee3f1aed2493e6549034d72ef3a5bcd39330b5c9f380b9531a" Oct 30 06:25:47.533552 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:25:47.533457 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:25:56.364443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:25:56.364404 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:26:13.456408 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:13.456368 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:26:17.497443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:17.497406 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:26:17.531140 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:26:17.531101 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:26:25.753905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.753865 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz"] Oct 30 06:26:25.758354 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.758327 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.794429 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.794397 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz"] Oct 30 06:26:25.898003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.897966 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898003 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898011 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898034 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898086 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898249 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898305 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898390 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c4c9\" (UniqueName: \"kubernetes.io/projected/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-kube-api-access-5c4c9\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898457 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.898511 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.898478 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999201 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999249 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c4c9\" (UniqueName: \"kubernetes.io/projected/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-kube-api-access-5c4c9\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999315 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999356 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999412 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999482 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999515 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999567 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:25.999942 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:25.999660 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.000076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.000050 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-credential-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.000223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.000198 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-workload-socket\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.000298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.000254 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-data\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.000298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.000273 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-workload-certs\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.000460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.000442 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.001987 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.001959 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-envoy\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.002204 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.002188 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-podinfo\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.025799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.025737 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c4c9\" (UniqueName: \"kubernetes.io/projected/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-kube-api-access-5c4c9\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.025799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.025753 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcf9bedc-ab39-4ca4-a56b-21db78fb6443-istio-token\") pod \"maas-default-gateway-openshift-default-84587756dc-6kkhz\" (UID: \"dcf9bedc-ab39-4ca4-a56b-21db78fb6443\") " pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.072217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.072184 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:26.393472 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.393410 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz"] Oct 30 06:26:26.395856 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:26:26.395830 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcf9bedc_ab39_4ca4_a56b_21db78fb6443.slice/crio-fc57660b63e229b1c42397341aad350d7dc71d75dd40252e35ff0917671e7dc9 WatchSource:0}: Error finding container fc57660b63e229b1c42397341aad350d7dc71d75dd40252e35ff0917671e7dc9: Status 404 returned error can't find the container with id fc57660b63e229b1c42397341aad350d7dc71d75dd40252e35ff0917671e7dc9 Oct 30 06:26:26.398028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.397996 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:26:26.398137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.398067 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:26:26.398137 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.398096 5565 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236068Ki","pods":"250"} Oct 30 06:26:26.560356 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.560312 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" event={"ID":"dcf9bedc-ab39-4ca4-a56b-21db78fb6443","Type":"ContainerStarted","Data":"5c4f0618c7d7c0b48ed93e5bbd530af227a02be9f30e4633c4c9708361b5e1da"} Oct 30 06:26:26.560607 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.560384 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" event={"ID":"dcf9bedc-ab39-4ca4-a56b-21db78fb6443","Type":"ContainerStarted","Data":"fc57660b63e229b1c42397341aad350d7dc71d75dd40252e35ff0917671e7dc9"} Oct 30 06:26:26.611191 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:26.611132 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podStartSLOduration=1.6111145470000001 podStartE2EDuration="1.611114547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:26:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:26:26.608827021 +0000 UTC m=+3293.184514077" watchObservedRunningTime="2026-10-30 06:26:26.611114547 +0000 UTC m=+3293.186801602" Oct 30 06:26:27.072989 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:27.072950 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:27.077549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:27.075173 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:27.077549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:27.075232 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:28.077554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:28.073952 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:28.077554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:28.074015 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:29.074551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:29.074008 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:29.074551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:29.074088 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:30.074553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:30.073953 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:30.074553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:30.074030 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:31.073554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:31.072996 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:31.073554 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:31.073076 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:32.076550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:32.075941 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:32.076550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:32.076006 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:33.073040 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:33.072999 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" start-of-body= Oct 30 06:26:33.073232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:33.073058 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": dial tcp 10.134.1.54:15021: connect: connection refused" Oct 30 06:26:33.506124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:33.506074 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:26:35.073832 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:35.073781 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:35.074265 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:35.073865 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:36.115776 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:36.115721 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:36.116351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:36.115848 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:37.116697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:37.116641 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:37.117257 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:37.116729 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:38.196553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:38.185658 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:38.196553 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:38.185750 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:39.232703 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.232666 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:26:39.233313 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.233278 5565 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:26:39.620350 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.620303 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:39.620765 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.620738 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:39.621623 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.621600 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" Oct 30 06:26:39.720789 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.720750 5565 scope.go:122] "RemoveContainer" containerID="1d8e7040cb396ae62ef1dda07d64d9aeb8890c8bcf30390d7a18ca4c39e76a1f" Oct 30 06:26:39.731422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.731398 5565 scope.go:122] "RemoveContainer" containerID="ff291cc86bea22f6f545b5ae8aae13b50d2eb937860c655b3891e9e66aa83d4d" Oct 30 06:26:39.748855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:39.748819 5565 scope.go:122] "RemoveContainer" containerID="fc6c28ebf4632bb7258fc3dcac237f7467fad17bc2cd868d11f250ed29aa66b7" Oct 30 06:26:47.535449 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:26:47.535404 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:26:49.165739 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:26:49.165698 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:27:12.200048 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:12.200010 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:27:17.530907 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:27:17.530864 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:27:22.655892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:22.655864 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:27:26.751933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:26.751899 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:27:39.785673 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:39.785636 5565 scope.go:122] "RemoveContainer" containerID="cb9262272a3bf2028dc5b96a4c2f15be0545442098caaae529ef6a7c91526dae" Oct 30 06:27:39.794156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:39.794133 5565 scope.go:122] "RemoveContainer" containerID="4bc9ae3e846cfb34979c8dedcd9552363285fe49cec5cb3b3b3d465cc37c354a" Oct 30 06:27:39.802067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:39.802046 5565 scope.go:122] "RemoveContainer" containerID="f400a7e545fe59b97827cca17b799a5a1ce8ba851d220c42434965cc83e3e939" Oct 30 06:27:44.200415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:44.200380 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:27:47.544950 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:27:47.544909 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:27:49.183136 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:49.183103 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:27:52.016066 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.016026 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k"] Oct 30 06:27:52.016517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.016240 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerName="discovery" containerID="cri-o://c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00" gracePeriod=30 Oct 30 06:27:52.280691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.280618 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 06:27:52.363725 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.363679 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.363946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.363767 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.363946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.363844 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.363946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.363905 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.364133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.363966 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.364133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.364016 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.364133 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.364042 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert\") pod \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\" (UID: \"78cd1982-682a-4f4b-9cba-3b7078cff5a2\") " Oct 30 06:27:52.364342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.364297 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:52.366467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.366428 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:52.366689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.366604 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:52.366689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.366641 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:52.366797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.366681 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:27:52.366797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.366700 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "kube-api-access-r26dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:52.366797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.366763 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert" pod "78cd1982-682a-4f4b-9cba-3b7078cff5a2" (UID: "78cd1982-682a-4f4b-9cba-3b7078cff5a2"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:52.465836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465789 5565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-ca-configmap\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:52.465836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465828 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r26dr\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-kube-api-access-r26dr\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:52.465836 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465842 5565 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/78cd1982-682a-4f4b-9cba-3b7078cff5a2-local-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:52.466113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465856 5565 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-kubeconfig\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:52.466113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465868 5565 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-csr-dns-cert\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:52.466113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465880 5565 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/78cd1982-682a-4f4b-9cba-3b7078cff5a2-istio-token\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:52.466113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:52.465890 5565 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/78cd1982-682a-4f4b-9cba-3b7078cff5a2-cacerts\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:53.063334 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.063301 5565 generic.go:356] "Generic (PLEG): container finished" podID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerID="c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00" exitCode=0 Oct 30 06:27:53.063875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.063352 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" Oct 30 06:27:53.063875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.063400 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" event={"ID":"78cd1982-682a-4f4b-9cba-3b7078cff5a2","Type":"ContainerDied","Data":"c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00"} Oct 30 06:27:53.063875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.063460 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" event={"ID":"78cd1982-682a-4f4b-9cba-3b7078cff5a2","Type":"ContainerDied","Data":"fadcee142e03c2f9a5965fa1438f910ffe1f308dc21a438088531234098e18a2"} Oct 30 06:27:53.063875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.063490 5565 scope.go:122] "RemoveContainer" containerID="c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00" Oct 30 06:27:53.076480 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.076456 5565 scope.go:122] "RemoveContainer" containerID="c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00" Oct 30 06:27:53.076802 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:27:53.076782 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00\": container with ID starting with c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00 not found: ID does not exist" containerID="c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00" Oct 30 06:27:53.076874 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.076813 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00"} err="failed to get container status \"c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00\": rpc error: code = NotFound desc = could not find container \"c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00\": container with ID starting with c9407c8b71ddc67e9ca5f28654b9af7a7f6458b55dab09d2ec08c06eca475c00 not found: ID does not exist" Oct 30 06:27:53.133396 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.133356 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k"] Oct 30 06:27:53.147247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.147213 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k"] Oct 30 06:27:53.565953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:53.565919 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" path="/var/lib/kubelet/pods/78cd1982-682a-4f4b-9cba-3b7078cff5a2/volumes" Oct 30 06:27:54.524933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.524895 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65"] Oct 30 06:27:54.525591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.525278 5565 cpu_manager.go:414] "containerMap: removing container" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerName="discovery" Oct 30 06:27:54.525591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.525289 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerName="discovery" Oct 30 06:27:54.525591 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.525388 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerName="discovery" Oct 30 06:27:54.529688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.529667 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.547016 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.546984 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:27:54.561031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.561005 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:27:54.563639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.563617 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65"] Oct 30 06:27:54.584095 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.584068 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:27:54.584291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.584268 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:27:54.591669 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.591647 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.591762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.591737 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.591887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.591872 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzj9v\" (UniqueName: \"kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.693042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.693010 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzj9v\" (UniqueName: \"kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.693222 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.693096 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.693222 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.693113 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.693749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.693723 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.695345 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.695328 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.717814 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.717792 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzj9v\" (UniqueName: \"kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v\") pod \"maas-api-revoke-sub-a4aef7d7-hqb65\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:54.847688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:54.847607 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:55.147636 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:55.147520 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65"] Oct 30 06:27:55.150370 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:27:55.150343 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda032508f_628a_4955_b560_ee0bee711a7c.slice/crio-1576d9e24c850c4e4e3d613809bc9330bd67d94c4bd525e5aa852089a8d7b8c1 WatchSource:0}: Error finding container 1576d9e24c850c4e4e3d613809bc9330bd67d94c4bd525e5aa852089a8d7b8c1: Status 404 returned error can't find the container with id 1576d9e24c850c4e4e3d613809bc9330bd67d94c4bd525e5aa852089a8d7b8c1 Oct 30 06:27:56.080961 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:56.080930 5565 generic.go:356] "Generic (PLEG): container finished" podID="a032508f-628a-4955-b560-ee0bee711a7c" containerID="c1d70356bffb4d38c88fb2d310bd2c0c528a5e1ec1ee3a0ec7e0689330a5e67d" exitCode=0 Oct 30 06:27:56.081326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:56.081019 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" event={"ID":"a032508f-628a-4955-b560-ee0bee711a7c","Type":"ContainerDied","Data":"c1d70356bffb4d38c88fb2d310bd2c0c528a5e1ec1ee3a0ec7e0689330a5e67d"} Oct 30 06:27:56.081326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:56.081090 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" event={"ID":"a032508f-628a-4955-b560-ee0bee711a7c","Type":"ContainerStarted","Data":"1576d9e24c850c4e4e3d613809bc9330bd67d94c4bd525e5aa852089a8d7b8c1"} Oct 30 06:27:57.198414 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.198381 5565 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-8s79k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:27:57.198763 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.198438 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-8s79k" podUID="78cd1982-682a-4f4b-9cba-3b7078cff5a2" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.105:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:27:57.212245 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.212220 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:57.317014 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.316989 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca\") pod \"a032508f-628a-4955-b560-ee0bee711a7c\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " Oct 30 06:27:57.317166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.317075 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca\") pod \"a032508f-628a-4955-b560-ee0bee711a7c\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " Oct 30 06:27:57.317166 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.317114 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v\" (UniqueName: \"kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v\") pod \"a032508f-628a-4955-b560-ee0bee711a7c\" (UID: \"a032508f-628a-4955-b560-ee0bee711a7c\") " Oct 30 06:27:57.317430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.317403 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca" pod "a032508f-628a-4955-b560-ee0bee711a7c" (UID: "a032508f-628a-4955-b560-ee0bee711a7c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:27:57.318994 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.318975 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca" pod "a032508f-628a-4955-b560-ee0bee711a7c" (UID: "a032508f-628a-4955-b560-ee0bee711a7c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:27:57.319226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.319203 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v" pod "a032508f-628a-4955-b560-ee0bee711a7c" (UID: "a032508f-628a-4955-b560-ee0bee711a7c"). InnerVolumeSpecName "kube-api-access-xzj9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:27:57.417932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.417873 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:57.417932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.417899 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xzj9v\" (UniqueName: \"kubernetes.io/projected/a032508f-628a-4955-b560-ee0bee711a7c-kube-api-access-xzj9v\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:57.417932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:57.417914 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a032508f-628a-4955-b560-ee0bee711a7c-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:27:58.091864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:58.091831 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" Oct 30 06:27:58.091864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:58.091855 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65" event={"ID":"a032508f-628a-4955-b560-ee0bee711a7c","Type":"ContainerDied","Data":"1576d9e24c850c4e4e3d613809bc9330bd67d94c4bd525e5aa852089a8d7b8c1"} Oct 30 06:27:58.092168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:27:58.091903 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1576d9e24c850c4e4e3d613809bc9330bd67d94c4bd525e5aa852089a8d7b8c1" Oct 30 06:28:05.699035 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:05.698996 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:28:10.182379 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:10.182326 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:28:12.682551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:12.681640 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-xp2kr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:28:12.682551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:12.681713 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-xp2kr" podUID="8611d5b8-cd8b-4117-b571-b96f2cde2621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:28:14.697857 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.697826 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5"] Oct 30 06:28:14.698496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.698473 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a032508f-628a-4955-b560-ee0bee711a7c" containerName="revoke-keys" Oct 30 06:28:14.698564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.698501 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a032508f-628a-4955-b560-ee0bee711a7c" containerName="revoke-keys" Oct 30 06:28:14.698666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.698652 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a032508f-628a-4955-b560-ee0bee711a7c" containerName="revoke-keys" Oct 30 06:28:14.701625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.701596 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.719141 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.719117 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:28:14.719394 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.719134 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:28:14.733126 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.733100 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:28:14.733322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.733305 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:28:14.733436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.733415 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkgtd\" (UniqueName: \"kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.733564 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.733547 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.733630 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.733613 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.734783 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.734761 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5"] Oct 30 06:28:14.834180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.834146 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.834363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.834218 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkgtd\" (UniqueName: \"kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.834363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.834333 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.835168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.835143 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.836639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.836606 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:14.859659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:14.859634 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkgtd\" (UniqueName: \"kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd\") pod \"maas-api-revoke-sub-79805ca9-5jxv5\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:15.015069 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:15.015037 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:15.314119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:15.314087 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5"] Oct 30 06:28:15.322884 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:28:15.322854 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e221f9e_8995_466c_8f2c_829b3eb3ae58.slice/crio-75c1f1ce598daac9786cd8993dcbf60fa42b664924b59ca3cd2ab279abeedffe WatchSource:0}: Error finding container 75c1f1ce598daac9786cd8993dcbf60fa42b664924b59ca3cd2ab279abeedffe: Status 404 returned error can't find the container with id 75c1f1ce598daac9786cd8993dcbf60fa42b664924b59ca3cd2ab279abeedffe Oct 30 06:28:16.214520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:16.214491 5565 generic.go:356] "Generic (PLEG): container finished" podID="0e221f9e-8995-466c-8f2c-829b3eb3ae58" containerID="d17db91fd341c96bd0390b387e7708afd6fce49b9dae7ef5f9c39c17efa0760a" exitCode=0 Oct 30 06:28:16.214948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:16.214572 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" event={"ID":"0e221f9e-8995-466c-8f2c-829b3eb3ae58","Type":"ContainerDied","Data":"d17db91fd341c96bd0390b387e7708afd6fce49b9dae7ef5f9c39c17efa0760a"} Oct 30 06:28:16.214948 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:16.214627 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" event={"ID":"0e221f9e-8995-466c-8f2c-829b3eb3ae58","Type":"ContainerStarted","Data":"75c1f1ce598daac9786cd8993dcbf60fa42b664924b59ca3cd2ab279abeedffe"} Oct 30 06:28:17.355466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.355441 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:17.458764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.458734 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca\") pod \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " Oct 30 06:28:17.458956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.458857 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca\") pod \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " Oct 30 06:28:17.458956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.458894 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd\" (UniqueName: \"kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd\") pod \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\" (UID: \"0e221f9e-8995-466c-8f2c-829b3eb3ae58\") " Oct 30 06:28:17.459156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.459132 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca" pod "0e221f9e-8995-466c-8f2c-829b3eb3ae58" (UID: "0e221f9e-8995-466c-8f2c-829b3eb3ae58"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:28:17.461055 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.461022 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd" pod "0e221f9e-8995-466c-8f2c-829b3eb3ae58" (UID: "0e221f9e-8995-466c-8f2c-829b3eb3ae58"). InnerVolumeSpecName "kube-api-access-lkgtd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:17.461168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.461060 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca" pod "0e221f9e-8995-466c-8f2c-829b3eb3ae58" (UID: "0e221f9e-8995-466c-8f2c-829b3eb3ae58"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:28:17.531825 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:28:17.531791 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:28:17.560717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.560686 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:17.560717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.560717 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lkgtd\" (UniqueName: \"kubernetes.io/projected/0e221f9e-8995-466c-8f2c-829b3eb3ae58-kube-api-access-lkgtd\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:17.560904 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:17.560732 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0e221f9e-8995-466c-8f2c-829b3eb3ae58-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:18.227073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:18.227039 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" event={"ID":"0e221f9e-8995-466c-8f2c-829b3eb3ae58","Type":"ContainerDied","Data":"75c1f1ce598daac9786cd8993dcbf60fa42b664924b59ca3cd2ab279abeedffe"} Oct 30 06:28:18.227073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:18.227070 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5" Oct 30 06:28:18.227284 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:18.227097 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75c1f1ce598daac9786cd8993dcbf60fa42b664924b59ca3cd2ab279abeedffe" Oct 30 06:28:25.810678 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:25.810643 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:28:30.045403 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:30.045363 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx"] Oct 30 06:28:30.069552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:30.064301 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a510e20-2tztx"] Oct 30 06:28:30.705551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:30.705483 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:28:31.565240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:31.565206 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="415f8bc5-30d5-45c0-a25e-648b7b3cfc26" path="/var/lib/kubelet/pods/415f8bc5-30d5-45c0-a25e-648b7b3cfc26/volumes" Oct 30 06:28:34.842207 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.842172 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps"] Oct 30 06:28:34.842612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.842593 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0e221f9e-8995-466c-8f2c-829b3eb3ae58" containerName="revoke-keys" Oct 30 06:28:34.842612 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.842607 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e221f9e-8995-466c-8f2c-829b3eb3ae58" containerName="revoke-keys" Oct 30 06:28:34.842707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.842676 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e221f9e-8995-466c-8f2c-829b3eb3ae58" containerName="revoke-keys" Oct 30 06:28:34.847012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.846995 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:34.878808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.878778 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:28:34.878947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.878804 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:28:34.878947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.878849 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:28:34.879077 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.878963 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:28:34.880683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.880660 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps"] Oct 30 06:28:34.960682 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.960661 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:34.960821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.960698 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:34.960821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:34.960720 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bp95\" (UniqueName: \"kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.061270 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.061246 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.061398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.061287 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.061398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.061317 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bp95\" (UniqueName: \"kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.061900 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.061881 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.063490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.063472 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.086515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.086495 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bp95\" (UniqueName: \"kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95\") pod \"maas-api-revoke-sub-0b7e8edb-69vps\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.159774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.159722 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:35.464819 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:35.464761 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps"] Oct 30 06:28:35.467636 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:28:35.467603 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1cd4b85_f6f1_4c16_a86f_5d787e86fd95.slice/crio-a04b650462cad0e80646cdb84bda61c9fbdbe37ef12e522cd0eeac15e676d544 WatchSource:0}: Error finding container a04b650462cad0e80646cdb84bda61c9fbdbe37ef12e522cd0eeac15e676d544: Status 404 returned error can't find the container with id a04b650462cad0e80646cdb84bda61c9fbdbe37ef12e522cd0eeac15e676d544 Oct 30 06:28:36.355099 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:36.355060 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" event={"ID":"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95","Type":"ContainerStarted","Data":"02707d8bbc682dbc344deadfac61c7f245e11ede43246c24fee396b84496f280"} Oct 30 06:28:36.355548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:36.355135 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" event={"ID":"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95","Type":"ContainerStarted","Data":"a04b650462cad0e80646cdb84bda61c9fbdbe37ef12e522cd0eeac15e676d544"} Oct 30 06:28:36.400754 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:36.400713 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" podStartSLOduration=2.400699104 podStartE2EDuration="2.400699104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:28:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:28:36.398882551 +0000 UTC m=+3422.974569606" watchObservedRunningTime="2026-10-30 06:28:36.400699104 +0000 UTC m=+3422.976386186" Oct 30 06:28:37.361349 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:37.361312 5565 generic.go:356] "Generic (PLEG): container finished" podID="c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" containerID="02707d8bbc682dbc344deadfac61c7f245e11ede43246c24fee396b84496f280" exitCode=0 Oct 30 06:28:37.361749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:37.361395 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" event={"ID":"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95","Type":"ContainerDied","Data":"02707d8bbc682dbc344deadfac61c7f245e11ede43246c24fee396b84496f280"} Oct 30 06:28:38.498439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.498415 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:38.688610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.688519 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95\" (UniqueName: \"kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95\") pod \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " Oct 30 06:28:38.688610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.688583 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca\") pod \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " Oct 30 06:28:38.688801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.688624 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca\") pod \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\" (UID: \"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95\") " Oct 30 06:28:38.689033 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.689012 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca" pod "c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" (UID: "c1cd4b85-f6f1-4c16-a86f-5d787e86fd95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:28:38.690683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.690648 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca" pod "c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" (UID: "c1cd4b85-f6f1-4c16-a86f-5d787e86fd95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:28:38.690794 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.690737 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95" pod "c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" (UID: "c1cd4b85-f6f1-4c16-a86f-5d787e86fd95"). InnerVolumeSpecName "kube-api-access-4bp95". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:38.789877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.789852 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bp95\" (UniqueName: \"kubernetes.io/projected/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-kube-api-access-4bp95\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:38.789972 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.789880 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:38.789972 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:38.789894 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:39.373163 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.373130 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" Oct 30 06:28:39.373327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.373129 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps" event={"ID":"c1cd4b85-f6f1-4c16-a86f-5d787e86fd95","Type":"ContainerDied","Data":"a04b650462cad0e80646cdb84bda61c9fbdbe37ef12e522cd0eeac15e676d544"} Oct 30 06:28:39.373327 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.373265 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a04b650462cad0e80646cdb84bda61c9fbdbe37ef12e522cd0eeac15e676d544" Oct 30 06:28:39.829801 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.829774 5565 scope.go:122] "RemoveContainer" containerID="3a0eb26f69623b0c18a285e6e107379a6cd70d2b22c1988295c6c2d8ea2d0a4b" Oct 30 06:28:39.838732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.838703 5565 scope.go:122] "RemoveContainer" containerID="cfeab602097c6477ee97bfd701d3e066ab5bef6fece5249d69ef9ecbbce13bd2" Oct 30 06:28:39.846907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.846886 5565 scope.go:122] "RemoveContainer" containerID="b1e34850a8ef2b7b8437c1b442713e2147962ee629d382215060f3221ac82fd3" Oct 30 06:28:39.854691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:39.854674 5565 scope.go:122] "RemoveContainer" containerID="6ef621e34cc43a472edf239642e529f431c36d59d597d346fdf5c06f4592d7be" Oct 30 06:28:45.647610 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.647508 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk"] Oct 30 06:28:45.648145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.648124 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" containerName="revoke-keys" Oct 30 06:28:45.648234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.648148 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" containerName="revoke-keys" Oct 30 06:28:45.648291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.648249 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" containerName="revoke-keys" Oct 30 06:28:45.653054 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.653031 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.685157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.685127 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:28:45.685157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.685128 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:28:45.685337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.685175 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:28:45.685337 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.685193 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:28:45.687853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.687832 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk"] Oct 30 06:28:45.741160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.741133 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zj5zn\" (UniqueName: \"kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.741338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.741189 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.741338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.741299 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.841797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.841772 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zj5zn\" (UniqueName: \"kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.841934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.841824 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.841934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.841851 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.842439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.842418 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.844169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.844141 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.867036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.867014 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zj5zn\" (UniqueName: \"kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn\") pod \"maas-api-revoke-sub-3d4156f1-877wk\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.968573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.968506 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:45.975597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:45.975567 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:28:46.272416 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:46.272373 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk"] Oct 30 06:28:46.274588 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:28:46.274563 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0782e09_34c4_4469_83e0_7a405f1ed6e0.slice/crio-2925bcc2cff3634358802edaf06b3168d9b26e150a10e1b6105d35973106bbd8 WatchSource:0}: Error finding container 2925bcc2cff3634358802edaf06b3168d9b26e150a10e1b6105d35973106bbd8: Status 404 returned error can't find the container with id 2925bcc2cff3634358802edaf06b3168d9b26e150a10e1b6105d35973106bbd8 Oct 30 06:28:46.411049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:46.411012 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" event={"ID":"a0782e09-34c4-4469-83e0-7a405f1ed6e0","Type":"ContainerStarted","Data":"2925bcc2cff3634358802edaf06b3168d9b26e150a10e1b6105d35973106bbd8"} Oct 30 06:28:47.417363 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:47.417274 5565 generic.go:356] "Generic (PLEG): container finished" podID="a0782e09-34c4-4469-83e0-7a405f1ed6e0" containerID="d59dfba6147cdb2dbffbe12e773feca87a5fc8074915d2b575cd5210b9b2f20f" exitCode=0 Oct 30 06:28:47.417851 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:47.417361 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" event={"ID":"a0782e09-34c4-4469-83e0-7a405f1ed6e0","Type":"ContainerDied","Data":"d59dfba6147cdb2dbffbe12e773feca87a5fc8074915d2b575cd5210b9b2f20f"} Oct 30 06:28:47.530579 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:28:47.530520 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:28:48.555407 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.555382 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:48.565820 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.565798 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca\") pod \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " Oct 30 06:28:48.565958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.565844 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca\") pod \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " Oct 30 06:28:48.565958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.565912 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn\" (UniqueName: \"kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn\") pod \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\" (UID: \"a0782e09-34c4-4469-83e0-7a405f1ed6e0\") " Oct 30 06:28:48.566265 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.566240 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca" pod "a0782e09-34c4-4469-83e0-7a405f1ed6e0" (UID: "a0782e09-34c4-4469-83e0-7a405f1ed6e0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:28:48.567768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.567738 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca" pod "a0782e09-34c4-4469-83e0-7a405f1ed6e0" (UID: "a0782e09-34c4-4469-83e0-7a405f1ed6e0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:28:48.567998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.567977 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn" pod "a0782e09-34c4-4469-83e0-7a405f1ed6e0" (UID: "a0782e09-34c4-4469-83e0-7a405f1ed6e0"). InnerVolumeSpecName "kube-api-access-zj5zn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:28:48.667223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.667196 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zj5zn\" (UniqueName: \"kubernetes.io/projected/a0782e09-34c4-4469-83e0-7a405f1ed6e0-kube-api-access-zj5zn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:48.667223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.667223 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:48.667406 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:48.667237 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0782e09-34c4-4469-83e0-7a405f1ed6e0-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:28:49.428614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:49.428574 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" Oct 30 06:28:49.428614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:49.428583 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk" event={"ID":"a0782e09-34c4-4469-83e0-7a405f1ed6e0","Type":"ContainerDied","Data":"2925bcc2cff3634358802edaf06b3168d9b26e150a10e1b6105d35973106bbd8"} Oct 30 06:28:49.428837 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:49.428633 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2925bcc2cff3634358802edaf06b3168d9b26e150a10e1b6105d35973106bbd8" Oct 30 06:28:56.861477 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:28:56.861445 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:29:02.139073 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:02.139042 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:29:05.927721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.927644 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk"] Oct 30 06:29:05.941616 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.941589 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-877wk"] Oct 30 06:29:05.979173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.979140 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl"] Oct 30 06:29:05.979571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.979553 5565 cpu_manager.go:414] "containerMap: removing container" podUID="a0782e09-34c4-4469-83e0-7a405f1ed6e0" containerName="revoke-keys" Oct 30 06:29:05.979571 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.979569 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0782e09-34c4-4469-83e0-7a405f1ed6e0" containerName="revoke-keys" Oct 30 06:29:05.979782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.979651 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0782e09-34c4-4469-83e0-7a405f1ed6e0" containerName="revoke-keys" Oct 30 06:29:05.983011 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:05.982990 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.002274 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.002245 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:29:06.002419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.002359 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:29:06.002419 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.002363 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:29:06.002619 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.002441 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:29:06.004226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.004205 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl"] Oct 30 06:29:06.029004 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.028970 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv"] Oct 30 06:29:06.034112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.034087 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9252fa77-cf4wv"] Oct 30 06:29:06.131177 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.131146 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwm2l\" (UniqueName: \"kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.131332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.131308 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.131380 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.131349 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.232746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.232679 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.232746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.232709 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.232746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.232735 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwm2l\" (UniqueName: \"kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.233415 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.233396 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.235089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.235072 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.257520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.257495 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwm2l\" (UniqueName: \"kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l\") pod \"maas-api-revoke-sub-3d4156f1-bs9jl\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.295450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.295429 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:06.596921 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:06.596896 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl"] Oct 30 06:29:06.599204 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:29:06.599178 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3196ae40_f4f1_45f0_8b77_d795b67f7599.slice/crio-c9970d7d891db8371be7779c8915e662d96b2284f8c43bc4a0d97f38562c20ab WatchSource:0}: Error finding container c9970d7d891db8371be7779c8915e662d96b2284f8c43bc4a0d97f38562c20ab: Status 404 returned error can't find the container with id c9970d7d891db8371be7779c8915e662d96b2284f8c43bc4a0d97f38562c20ab Oct 30 06:29:07.560973 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:07.560944 5565 generic.go:356] "Generic (PLEG): container finished" podID="3196ae40-f4f1-45f0-8b77-d795b67f7599" containerID="818f96474e77edaee20eff1bc7d3c0f1d1b6324c2ad0bb9a184dd11ee4ece21e" exitCode=0 Oct 30 06:29:07.565181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:07.565158 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39c60b43-61b5-4368-9e39-d90264f28536" path="/var/lib/kubelet/pods/39c60b43-61b5-4368-9e39-d90264f28536/volumes" Oct 30 06:29:07.565558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:07.565545 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0782e09-34c4-4469-83e0-7a405f1ed6e0" path="/var/lib/kubelet/pods/a0782e09-34c4-4469-83e0-7a405f1ed6e0/volumes" Oct 30 06:29:07.565802 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:07.565786 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" event={"ID":"3196ae40-f4f1-45f0-8b77-d795b67f7599","Type":"ContainerDied","Data":"818f96474e77edaee20eff1bc7d3c0f1d1b6324c2ad0bb9a184dd11ee4ece21e"} Oct 30 06:29:07.565844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:07.565825 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" event={"ID":"3196ae40-f4f1-45f0-8b77-d795b67f7599","Type":"ContainerStarted","Data":"c9970d7d891db8371be7779c8915e662d96b2284f8c43bc4a0d97f38562c20ab"} Oct 30 06:29:08.695031 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.695005 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:08.854975 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.854901 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca\") pod \"3196ae40-f4f1-45f0-8b77-d795b67f7599\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " Oct 30 06:29:08.855119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.855085 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca\") pod \"3196ae40-f4f1-45f0-8b77-d795b67f7599\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " Oct 30 06:29:08.855119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.855105 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l\" (UniqueName: \"kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l\") pod \"3196ae40-f4f1-45f0-8b77-d795b67f7599\" (UID: \"3196ae40-f4f1-45f0-8b77-d795b67f7599\") " Oct 30 06:29:08.855504 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.855474 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca" pod "3196ae40-f4f1-45f0-8b77-d795b67f7599" (UID: "3196ae40-f4f1-45f0-8b77-d795b67f7599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:08.856959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.856934 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca" pod "3196ae40-f4f1-45f0-8b77-d795b67f7599" (UID: "3196ae40-f4f1-45f0-8b77-d795b67f7599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:08.857084 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.857066 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l" pod "3196ae40-f4f1-45f0-8b77-d795b67f7599" (UID: "3196ae40-f4f1-45f0-8b77-d795b67f7599"). InnerVolumeSpecName "kube-api-access-rwm2l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:08.955873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.955844 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:29:08.955873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.955870 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3196ae40-f4f1-45f0-8b77-d795b67f7599-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:29:08.956025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:08.955884 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwm2l\" (UniqueName: \"kubernetes.io/projected/3196ae40-f4f1-45f0-8b77-d795b67f7599-kube-api-access-rwm2l\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:29:09.577934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:09.577904 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" Oct 30 06:29:09.578080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:09.577897 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl" event={"ID":"3196ae40-f4f1-45f0-8b77-d795b67f7599","Type":"ContainerDied","Data":"c9970d7d891db8371be7779c8915e662d96b2284f8c43bc4a0d97f38562c20ab"} Oct 30 06:29:09.578080 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:09.578028 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c9970d7d891db8371be7779c8915e662d96b2284f8c43bc4a0d97f38562c20ab" Oct 30 06:29:17.531910 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:29:17.531873 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:29:17.853394 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:17.853317 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:29:21.027314 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:21.027282 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45"] Oct 30 06:29:21.041232 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:21.041207 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-056eac1f-62n45"] Oct 30 06:29:21.565505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:21.565473 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a13018c-df26-401e-adaa-0a4bdd97e52a" path="/var/lib/kubelet/pods/7a13018c-df26-401e-adaa-0a4bdd97e52a/volumes" Oct 30 06:29:22.383684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:22.383652 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:29:28.033993 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:28.033956 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl"] Oct 30 06:29:28.047775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:28.047741 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af64ac97-w97hl"] Oct 30 06:29:29.544128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.544093 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl"] Oct 30 06:29:29.558076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.558052 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bs9jl"] Oct 30 06:29:29.565467 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.565443 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3196ae40-f4f1-45f0-8b77-d795b67f7599" path="/var/lib/kubelet/pods/3196ae40-f4f1-45f0-8b77-d795b67f7599/volumes" Oct 30 06:29:29.565878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.565864 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99" path="/var/lib/kubelet/pods/b2bf6f0d-9bc7-4d7b-a19c-5d1f06e62f99/volumes" Oct 30 06:29:29.585413 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.585390 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr"] Oct 30 06:29:29.585793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.585778 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3196ae40-f4f1-45f0-8b77-d795b67f7599" containerName="revoke-keys" Oct 30 06:29:29.585841 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.585794 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3196ae40-f4f1-45f0-8b77-d795b67f7599" containerName="revoke-keys" Oct 30 06:29:29.585896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.585887 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3196ae40-f4f1-45f0-8b77-d795b67f7599" containerName="revoke-keys" Oct 30 06:29:29.590013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.589998 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.607235 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.607206 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:29:29.607494 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.607466 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:29:29.607617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.607545 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:29:29.607686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.607643 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:29:29.609601 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.609577 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr"] Oct 30 06:29:29.728300 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.728272 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.728667 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.728641 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpnnx\" (UniqueName: \"kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.728748 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.728734 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.829831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.829763 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.829952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.829833 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpnnx\" (UniqueName: \"kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.829952 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.829863 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.830430 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.830410 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.832065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.832042 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.854298 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.854272 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpnnx\" (UniqueName: \"kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx\") pod \"maas-api-revoke-sub-3d4156f1-bdxbr\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:29.903119 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:29.903090 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:30.204140 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:30.204113 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr"] Oct 30 06:29:30.205692 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:29:30.205660 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6238ddcb_2d47_4057_90f4_bf555049dcb9.slice/crio-48ef8a8a7f89869c7f54a0460b334a67cb0e34d032439e56f87faceb3e3c3417 WatchSource:0}: Error finding container 48ef8a8a7f89869c7f54a0460b334a67cb0e34d032439e56f87faceb3e3c3417: Status 404 returned error can't find the container with id 48ef8a8a7f89869c7f54a0460b334a67cb0e34d032439e56f87faceb3e3c3417 Oct 30 06:29:30.680945 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:30.680905 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" event={"ID":"6238ddcb-2d47-4057-90f4-bf555049dcb9","Type":"ContainerStarted","Data":"4c7a9602dc4ac559f84fbab4f4696de09d72084812a0ce9920be57490f384e5f"} Oct 30 06:29:30.681288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:30.680966 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" event={"ID":"6238ddcb-2d47-4057-90f4-bf555049dcb9","Type":"ContainerStarted","Data":"48ef8a8a7f89869c7f54a0460b334a67cb0e34d032439e56f87faceb3e3c3417"} Oct 30 06:29:30.725065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:30.724976 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" podStartSLOduration=1.724960617 podStartE2EDuration="1.724960617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:30.724899606 +0000 UTC m=+3477.300586664" watchObservedRunningTime="2026-10-30 06:29:30.724960617 +0000 UTC m=+3477.300647672" Oct 30 06:29:31.686892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:31.686856 5565 generic.go:356] "Generic (PLEG): container finished" podID="6238ddcb-2d47-4057-90f4-bf555049dcb9" containerID="4c7a9602dc4ac559f84fbab4f4696de09d72084812a0ce9920be57490f384e5f" exitCode=0 Oct 30 06:29:31.687292 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:31.686935 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" event={"ID":"6238ddcb-2d47-4057-90f4-bf555049dcb9","Type":"ContainerDied","Data":"4c7a9602dc4ac559f84fbab4f4696de09d72084812a0ce9920be57490f384e5f"} Oct 30 06:29:32.821045 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.821019 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:32.958338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.958274 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx\" (UniqueName: \"kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx\") pod \"6238ddcb-2d47-4057-90f4-bf555049dcb9\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " Oct 30 06:29:32.958482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.958338 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca\") pod \"6238ddcb-2d47-4057-90f4-bf555049dcb9\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " Oct 30 06:29:32.958482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.958396 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca\") pod \"6238ddcb-2d47-4057-90f4-bf555049dcb9\" (UID: \"6238ddcb-2d47-4057-90f4-bf555049dcb9\") " Oct 30 06:29:32.958815 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.958790 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca" pod "6238ddcb-2d47-4057-90f4-bf555049dcb9" (UID: "6238ddcb-2d47-4057-90f4-bf555049dcb9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:29:32.960386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.960365 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca" pod "6238ddcb-2d47-4057-90f4-bf555049dcb9" (UID: "6238ddcb-2d47-4057-90f4-bf555049dcb9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:29:32.960496 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:32.960426 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx" pod "6238ddcb-2d47-4057-90f4-bf555049dcb9" (UID: "6238ddcb-2d47-4057-90f4-bf555049dcb9"). InnerVolumeSpecName "kube-api-access-dpnnx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:29:33.059362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:33.059332 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpnnx\" (UniqueName: \"kubernetes.io/projected/6238ddcb-2d47-4057-90f4-bf555049dcb9-kube-api-access-dpnnx\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:29:33.059362 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:33.059361 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:29:33.059573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:33.059376 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6238ddcb-2d47-4057-90f4-bf555049dcb9-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:29:33.697556 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:33.697514 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" Oct 30 06:29:33.697709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:33.697557 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr" event={"ID":"6238ddcb-2d47-4057-90f4-bf555049dcb9","Type":"ContainerDied","Data":"48ef8a8a7f89869c7f54a0460b334a67cb0e34d032439e56f87faceb3e3c3417"} Oct 30 06:29:33.697709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:33.697600 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48ef8a8a7f89869c7f54a0460b334a67cb0e34d032439e56f87faceb3e3c3417" Oct 30 06:29:39.888614 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:39.888585 5565 scope.go:122] "RemoveContainer" containerID="9dfbed8469979e7a6b14b82d89ed5b58729c88ab1e37594c980a458236a8f2c8" Oct 30 06:29:39.897509 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:39.897486 5565 scope.go:122] "RemoveContainer" containerID="fced9fd236e7bd4a9488353be02aecfc65575038225ec0a33670610173e9ff06" Oct 30 06:29:39.905352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:39.905330 5565 scope.go:122] "RemoveContainer" containerID="514e41d9dbc5e182645f812b51d1f4cc3ababb3b3404c3bb2e7c09095cbee4aa" Oct 30 06:29:39.913338 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:39.913320 5565 scope.go:122] "RemoveContainer" containerID="f81ff4c1212010b0f546d69da7eab545b7d52f3fc3491b12cee673da0ba4ae52" Oct 30 06:29:39.922047 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:39.922023 5565 scope.go:122] "RemoveContainer" containerID="ab09e7d2488e58909661e7b79fdba4a3b221726b649c5b4e483879c701958844" Oct 30 06:29:39.929701 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:39.929685 5565 scope.go:122] "RemoveContainer" containerID="1c256d0f4ef47616a8d710e23aee5378be2bd7e3290c05723bfe88065420fef3" Oct 30 06:29:41.488222 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:41.488190 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:29:46.195317 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:46.195282 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:29:47.531916 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:29:47.531878 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:29:56.344492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.344459 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz"] Oct 30 06:29:56.344889 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.344875 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6238ddcb-2d47-4057-90f4-bf555049dcb9" containerName="revoke-keys" Oct 30 06:29:56.344889 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.344888 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6238ddcb-2d47-4057-90f4-bf555049dcb9" containerName="revoke-keys" Oct 30 06:29:56.344966 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.344949 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6238ddcb-2d47-4057-90f4-bf555049dcb9" containerName="revoke-keys" Oct 30 06:29:56.350214 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.350186 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.383438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.383406 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:29:56.383702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.383675 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:29:56.383919 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.383895 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-627222-serving-cert\"" Oct 30 06:29:56.386260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.386228 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz"] Oct 30 06:29:56.386450 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.386429 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:29:56.386558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.386460 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wbd8\"" Oct 30 06:29:56.404560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.404522 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g"] Oct 30 06:29:56.408639 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.408617 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.433939 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.433911 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g"] Oct 30 06:29:56.457038 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.457003 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp"] Oct 30 06:29:56.461108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.461067 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.461956 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.461919 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.462086 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.462050 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.462195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.462082 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlh8w\" (UniqueName: \"kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w\") pod \"payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.462195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.462143 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.462195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.462178 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.462374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.462222 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfgbf\" (UniqueName: \"kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.479506 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.479481 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-627222\"" Oct 30 06:29:56.479718 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.479700 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3ee2313c-dockercfg-4ms9d\"" Oct 30 06:29:56.486857 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.486830 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp"] Oct 30 06:29:56.563001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.562963 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.563156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563013 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlh8w\" (UniqueName: \"kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w\") pod \"payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.563156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563061 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.563156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563095 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.563156 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563132 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.563417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563214 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfgbf\" (UniqueName: \"kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.563417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563277 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.563417 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563329 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wl22c\" (UniqueName: \"kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c\") pod \"payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.563953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.563926 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.564142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.564115 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.566219 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.566197 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.573043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.573013 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.589180 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.589146 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlh8w\" (UniqueName: \"kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w\") pod \"payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.589342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.589292 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfgbf\" (UniqueName: \"kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf\") pod \"maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.663493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.663426 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:56.663900 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.663874 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.663953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.663934 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wl22c\" (UniqueName: \"kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c\") pod \"payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.664679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.664654 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.692751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.692721 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wl22c\" (UniqueName: \"kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c\") pod \"payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.726675 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.726637 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:56.774470 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.774431 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:56.985558 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.985516 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz"] Oct 30 06:29:56.987660 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:29:56.987623 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd00a5f_6e8d_4156_ac45_3fac84a461b2.slice/crio-feb87644278baa451b6637526e1d7556457dce02b0e769de3b6b4d0ad7bf0a49 WatchSource:0}: Error finding container feb87644278baa451b6637526e1d7556457dce02b0e769de3b6b4d0ad7bf0a49: Status 404 returned error can't find the container with id feb87644278baa451b6637526e1d7556457dce02b0e769de3b6b4d0ad7bf0a49 Oct 30 06:29:56.989092 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:56.989070 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:29:57.033907 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.033883 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g"] Oct 30 06:29:57.036889 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:29:57.036866 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc04e21e0_afaa_4dd6_bf24_936d5a30438b.slice/crio-d57c82fddf06b5eb792e008d558c71c7fb55c9f985befe7fa9574a6a5ad4dbe4 WatchSource:0}: Error finding container d57c82fddf06b5eb792e008d558c71c7fb55c9f985befe7fa9574a6a5ad4dbe4: Status 404 returned error can't find the container with id d57c82fddf06b5eb792e008d558c71c7fb55c9f985befe7fa9574a6a5ad4dbe4 Oct 30 06:29:57.086083 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.086058 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp"] Oct 30 06:29:57.087835 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:29:57.087814 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056f5b7e_883c_4898_ad98_a0834c18b154.slice/crio-a1d7576579de95ba33e1c08e10bb3bebc48f531d79414e5988809c5122b1ff75 WatchSource:0}: Error finding container a1d7576579de95ba33e1c08e10bb3bebc48f531d79414e5988809c5122b1ff75: Status 404 returned error can't find the container with id a1d7576579de95ba33e1c08e10bb3bebc48f531d79414e5988809c5122b1ff75 Oct 30 06:29:57.813422 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.813389 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" event={"ID":"056f5b7e-883c-4898-ad98-a0834c18b154","Type":"ContainerStarted","Data":"401f3412b1a1d72732f2442eb6d53d9868fb3a7cbd442b5c202359038ab6f006"} Oct 30 06:29:57.813902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.813438 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" event={"ID":"056f5b7e-883c-4898-ad98-a0834c18b154","Type":"ContainerStarted","Data":"a1d7576579de95ba33e1c08e10bb3bebc48f531d79414e5988809c5122b1ff75"} Oct 30 06:29:57.813902 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.813511 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:29:57.815154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.815129 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" event={"ID":"c04e21e0-afaa-4dd6-bf24-936d5a30438b","Type":"ContainerStarted","Data":"92f15f9a517ecb2990ff04de5bf516e4ff9b4f254b11e627f42d6a2e0459b43e"} Oct 30 06:29:57.815294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.815165 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" event={"ID":"c04e21e0-afaa-4dd6-bf24-936d5a30438b","Type":"ContainerStarted","Data":"d57c82fddf06b5eb792e008d558c71c7fb55c9f985befe7fa9574a6a5ad4dbe4"} Oct 30 06:29:57.815294 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.815199 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:29:57.816500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.816479 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" event={"ID":"5cd00a5f-6e8d-4156-ac45-3fac84a461b2","Type":"ContainerStarted","Data":"119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8"} Oct 30 06:29:57.816617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.816522 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" event={"ID":"5cd00a5f-6e8d-4156-ac45-3fac84a461b2","Type":"ContainerStarted","Data":"feb87644278baa451b6637526e1d7556457dce02b0e769de3b6b4d0ad7bf0a49"} Oct 30 06:29:57.816617 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.816589 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:29:57.858886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.858845 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" podStartSLOduration=1.858834119 podStartE2EDuration="1.858834119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:57.857514281 +0000 UTC m=+3504.433201335" watchObservedRunningTime="2026-10-30 06:29:57.858834119 +0000 UTC m=+3504.434521176" Oct 30 06:29:57.908281 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.908240 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" podStartSLOduration=1.5709024299999998 podStartE2EDuration="1.908228055s" totalImagesPullingTime="337.325625ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:57.907709615 +0000 UTC m=+3504.483396669" watchObservedRunningTime="2026-10-30 06:29:57.908228055 +0000 UTC m=+3504.483915108" Oct 30 06:29:57.952600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:57.952559 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" podStartSLOduration=1.952546339 podStartE2EDuration="1.952546339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:29:57.951366795 +0000 UTC m=+3504.527053849" watchObservedRunningTime="2026-10-30 06:29:57.952546339 +0000 UTC m=+3504.528233392" Oct 30 06:29:59.669246 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:29:59.669210 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:30:00.147686 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.147657 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z"] Oct 30 06:30:00.150982 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.150963 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.168968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.168945 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:30:00.184602 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.184576 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z"] Oct 30 06:30:00.299469 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.299434 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.299663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.299492 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csl6j\" (UniqueName: \"kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j\") pod \"maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.400799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.400721 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.400799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.400777 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csl6j\" (UniqueName: \"kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j\") pod \"maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.401349 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.401328 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.426227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.426190 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-csl6j\" (UniqueName: \"kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j\") pod \"maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.466682 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.466657 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:00.770213 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.770186 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z"] Oct 30 06:30:00.772551 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:30:00.772500 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef401418_87c5_4ca7_bf26_64660c5a6545.slice/crio-a9e7f35efa3fa65b5122a63f95cdb6fc0679b91f54c54aee62454352633624b9 WatchSource:0}: Error finding container a9e7f35efa3fa65b5122a63f95cdb6fc0679b91f54c54aee62454352633624b9: Status 404 returned error can't find the container with id a9e7f35efa3fa65b5122a63f95cdb6fc0679b91f54c54aee62454352633624b9 Oct 30 06:30:00.841517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:00.841487 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" event={"ID":"ef401418-87c5-4ca7-bf26-64660c5a6545","Type":"ContainerStarted","Data":"a9e7f35efa3fa65b5122a63f95cdb6fc0679b91f54c54aee62454352633624b9"} Oct 30 06:30:01.847795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:01.847710 5565 generic.go:356] "Generic (PLEG): container finished" podID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerID="9d2ce5dff006d14df5f3084a88b996d8f719243317a9d5a2a8bcf6504242fa4a" exitCode=7 Oct 30 06:30:01.848258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:01.847793 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" event={"ID":"ef401418-87c5-4ca7-bf26-64660c5a6545","Type":"ContainerDied","Data":"9d2ce5dff006d14df5f3084a88b996d8f719243317a9d5a2a8bcf6504242fa4a"} Oct 30 06:30:01.848258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:01.848018 5565 scope.go:122] "RemoveContainer" containerID="9d2ce5dff006d14df5f3084a88b996d8f719243317a9d5a2a8bcf6504242fa4a" Oct 30 06:30:02.853977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:02.853896 5565 generic.go:356] "Generic (PLEG): container finished" podID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerID="f0a5207435cf3ed633c7a199d1ce84f154cfa7113e530576b95e7baf4c475b9e" exitCode=7 Oct 30 06:30:02.854395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:02.853970 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" event={"ID":"ef401418-87c5-4ca7-bf26-64660c5a6545","Type":"ContainerDied","Data":"f0a5207435cf3ed633c7a199d1ce84f154cfa7113e530576b95e7baf4c475b9e"} Oct 30 06:30:02.854395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:02.854036 5565 scope.go:122] "RemoveContainer" containerID="9d2ce5dff006d14df5f3084a88b996d8f719243317a9d5a2a8bcf6504242fa4a" Oct 30 06:30:02.854395 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:02.854268 5565 scope.go:122] "RemoveContainer" containerID="f0a5207435cf3ed633c7a199d1ce84f154cfa7113e530576b95e7baf4c475b9e" Oct 30 06:30:02.854618 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:30:02.854476 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z_odh-ai-gateway-infra(ef401418-87c5-4ca7-bf26-64660c5a6545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" Oct 30 06:30:03.826524 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:03.826494 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:30:03.860697 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:03.860674 5565 scope.go:122] "RemoveContainer" containerID="f0a5207435cf3ed633c7a199d1ce84f154cfa7113e530576b95e7baf4c475b9e" Oct 30 06:30:03.861037 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:30:03.860872 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z_odh-ai-gateway-infra(ef401418-87c5-4ca7-bf26-64660c5a6545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" Oct 30 06:30:08.823931 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:08.823900 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:30:08.824342 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:08.823965 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:30:14.717983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:14.693675 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:30:16.559957 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:16.559930 5565 scope.go:122] "RemoveContainer" containerID="f0a5207435cf3ed633c7a199d1ce84f154cfa7113e530576b95e7baf4c475b9e" Oct 30 06:30:16.932877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:16.932840 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" event={"ID":"ef401418-87c5-4ca7-bf26-64660c5a6545","Type":"ContainerStarted","Data":"0f1ee0a730bc42566bac6876db54380903b8250100888f7816dbe6c5e2cc3bb3"} Oct 30 06:30:16.980754 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:16.980709 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" podStartSLOduration=16.980693061 podStartE2EDuration="16.980693061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:30:16.979629139 +0000 UTC m=+3523.555316193" watchObservedRunningTime="2026-10-30 06:30:16.980693061 +0000 UTC m=+3523.556380154" Oct 30 06:30:17.532595 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:30:17.532486 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:30:17.939573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:17.939479 5565 generic.go:356] "Generic (PLEG): container finished" podID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerID="0f1ee0a730bc42566bac6876db54380903b8250100888f7816dbe6c5e2cc3bb3" exitCode=60 Oct 30 06:30:17.939573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:17.939552 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" event={"ID":"ef401418-87c5-4ca7-bf26-64660c5a6545","Type":"ContainerDied","Data":"0f1ee0a730bc42566bac6876db54380903b8250100888f7816dbe6c5e2cc3bb3"} Oct 30 06:30:17.940065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:17.939612 5565 scope.go:122] "RemoveContainer" containerID="f0a5207435cf3ed633c7a199d1ce84f154cfa7113e530576b95e7baf4c475b9e" Oct 30 06:30:17.940065 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:17.939873 5565 scope.go:122] "RemoveContainer" containerID="0f1ee0a730bc42566bac6876db54380903b8250100888f7816dbe6c5e2cc3bb3" Oct 30 06:30:17.940169 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:30:17.940091 5565 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z_odh-ai-gateway-infra(ef401418-87c5-4ca7-bf26-64660c5a6545)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" Oct 30 06:30:17.990800 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:17.990775 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z"] Oct 30 06:30:19.078778 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.078753 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:19.095840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.095817 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca\") pod \"ef401418-87c5-4ca7-bf26-64660c5a6545\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " Oct 30 06:30:19.096006 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.095990 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j\" (UniqueName: \"kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j\") pod \"ef401418-87c5-4ca7-bf26-64660c5a6545\" (UID: \"ef401418-87c5-4ca7-bf26-64660c5a6545\") " Oct 30 06:30:19.096142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.096120 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca" pod "ef401418-87c5-4ca7-bf26-64660c5a6545" (UID: "ef401418-87c5-4ca7-bf26-64660c5a6545"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:19.096330 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.096310 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef401418-87c5-4ca7-bf26-64660c5a6545-maas-api-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.098019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.097994 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j" pod "ef401418-87c5-4ca7-bf26-64660c5a6545" (UID: "ef401418-87c5-4ca7-bf26-64660c5a6545"). InnerVolumeSpecName "kube-api-access-csl6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:19.197873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.197795 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-csl6j\" (UniqueName: \"kubernetes.io/projected/ef401418-87c5-4ca7-bf26-64660c5a6545-kube-api-access-csl6j\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:30:19.952713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.952624 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" Oct 30 06:30:19.952713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.952639 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z" event={"ID":"ef401418-87c5-4ca7-bf26-64660c5a6545","Type":"ContainerDied","Data":"a9e7f35efa3fa65b5122a63f95cdb6fc0679b91f54c54aee62454352633624b9"} Oct 30 06:30:19.952951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:19.952734 5565 scope.go:122] "RemoveContainer" containerID="0f1ee0a730bc42566bac6876db54380903b8250100888f7816dbe6c5e2cc3bb3" Oct 30 06:30:20.009637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:20.009607 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z"] Oct 30 06:30:20.023689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:20.023651 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-single-main-627222-2988903sww2z"] Oct 30 06:30:21.565272 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:21.565242 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" path="/var/lib/kubelet/pods/ef401418-87c5-4ca7-bf26-64660c5a6545/volumes" Oct 30 06:30:29.266239 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:29.266205 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:30:39.972307 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:39.972273 5565 scope.go:122] "RemoveContainer" containerID="62b27f1dea0838713ca4e65445642fb1d5f3495de7701532cda04aae9556c10e" Oct 30 06:30:39.981049 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:39.981016 5565 scope.go:122] "RemoveContainer" containerID="b4f824e897fb46a5686500b4a726fcd9225de648a6c75d06ed2760e185f7ac33" Oct 30 06:30:39.988805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:39.988783 5565 scope.go:122] "RemoveContainer" containerID="ed75cd69896176e16037cad184e3e0bf6cbe7578927d095fd54069eca8cb0d63" Oct 30 06:30:46.695561 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:46.695507 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:30:47.531204 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:30:47.531162 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:30:57.019514 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.019477 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g"] Oct 30 06:30:57.051878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.051839 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp"] Oct 30 06:30:57.448098 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.448012 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz"] Oct 30 06:30:57.448306 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.448281 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" podUID="5cd00a5f-6e8d-4156-ac45-3fac84a461b2" containerName="maas-api" containerID="cri-o://119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8" gracePeriod=30 Oct 30 06:30:57.912172 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.912137 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:30:57.952355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.952306 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls\") pod \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " Oct 30 06:30:57.952589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.952505 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca\") pod \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " Oct 30 06:30:57.952589 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.952574 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs\") pod \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " Oct 30 06:30:57.952722 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.952604 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf\" (UniqueName: \"kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf\") pod \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\" (UID: \"5cd00a5f-6e8d-4156-ac45-3fac84a461b2\") " Oct 30 06:30:57.953200 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.953165 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca" pod "5cd00a5f-6e8d-4156-ac45-3fac84a461b2" (UID: "5cd00a5f-6e8d-4156-ac45-3fac84a461b2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:30:57.954710 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.954681 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls" pod "5cd00a5f-6e8d-4156-ac45-3fac84a461b2" (UID: "5cd00a5f-6e8d-4156-ac45-3fac84a461b2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:57.954808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.954708 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf" pod "5cd00a5f-6e8d-4156-ac45-3fac84a461b2" (UID: "5cd00a5f-6e8d-4156-ac45-3fac84a461b2"). InnerVolumeSpecName "kube-api-access-xfgbf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:30:57.954808 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:57.954776 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs" pod "5cd00a5f-6e8d-4156-ac45-3fac84a461b2" (UID: "5cd00a5f-6e8d-4156-ac45-3fac84a461b2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:30:58.054251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.054211 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:30:58.054251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.054246 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:30:58.054251 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.054257 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:30:58.054734 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.054267 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfgbf\" (UniqueName: \"kubernetes.io/projected/5cd00a5f-6e8d-4156-ac45-3fac84a461b2-kube-api-access-xfgbf\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:30:58.138611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.138576 5565 generic.go:356] "Generic (PLEG): container finished" podID="5cd00a5f-6e8d-4156-ac45-3fac84a461b2" containerID="119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8" exitCode=0 Oct 30 06:30:58.138812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.138621 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" Oct 30 06:30:58.138812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.138664 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" event={"ID":"5cd00a5f-6e8d-4156-ac45-3fac84a461b2","Type":"ContainerDied","Data":"119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8"} Oct 30 06:30:58.138812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.138717 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz" event={"ID":"5cd00a5f-6e8d-4156-ac45-3fac84a461b2","Type":"ContainerDied","Data":"feb87644278baa451b6637526e1d7556457dce02b0e769de3b6b4d0ad7bf0a49"} Oct 30 06:30:58.138812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.138731 5565 scope.go:122] "RemoveContainer" containerID="119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8" Oct 30 06:30:58.148352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.148336 5565 scope.go:122] "RemoveContainer" containerID="119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8" Oct 30 06:30:58.148651 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:30:58.148625 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8\": container with ID starting with 119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8 not found: ID does not exist" containerID="119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8" Oct 30 06:30:58.148762 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.148654 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8"} err="failed to get container status \"119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8\": rpc error: code = NotFound desc = could not find container \"119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8\": container with ID starting with 119c44e7d7bb73a41fd0ee8fd138ab156fd46d001ba106bfdb7f0481e38b6cc8 not found: ID does not exist" Oct 30 06:30:58.195044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.195021 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz"] Oct 30 06:30:58.201777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:58.201748 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-627222-75d9f67464-j85zz"] Oct 30 06:30:59.565798 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:59.565764 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd00a5f-6e8d-4156-ac45-3fac84a461b2" path="/var/lib/kubelet/pods/5cd00a5f-6e8d-4156-ac45-3fac84a461b2/volumes" Oct 30 06:30:59.593322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:30:59.593290 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:31:02.020193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:02.020149 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerName="payload-processing" containerID="cri-o://92f15f9a517ecb2990ff04de5bf516e4ff9b4f254b11e627f42d6a2e0459b43e" gracePeriod=25 Oct 30 06:31:02.052934 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:02.052885 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" containerName="payload-pre-processing" containerID="cri-o://401f3412b1a1d72732f2442eb6d53d9868fb3a7cbd442b5c202359038ab6f006" gracePeriod=25 Oct 30 06:31:08.105827 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.105772 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr"] Oct 30 06:31:08.119677 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.119640 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-bdxbr"] Oct 30 06:31:08.156071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156040 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr"] Oct 30 06:31:08.156555 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156522 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156558 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156571 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156576 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156594 5565 cpu_manager.go:414] "containerMap: removing container" podUID="5cd00a5f-6e8d-4156-ac45-3fac84a461b2" containerName="maas-api" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156599 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd00a5f-6e8d-4156-ac45-3fac84a461b2" containerName="maas-api" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156608 5565 cpu_manager.go:414] "containerMap: removing container" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.156666 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156614 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.157027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156684 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.157027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156692 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:31:08.157027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.156701 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd00a5f-6e8d-4156-ac45-3fac84a461b2" containerName="maas-api" Oct 30 06:31:08.161637 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.161612 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.179381 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.178570 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:31:08.179381 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.178580 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:31:08.179683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.179495 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:31:08.179683 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.179546 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:31:08.180128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.180102 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr"] Oct 30 06:31:08.254643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.254597 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.254643 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.254650 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl7kg\" (UniqueName: \"kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.254894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.254791 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.355368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.355323 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.355368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.355371 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl7kg\" (UniqueName: \"kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.355631 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.355430 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.356192 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.356134 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.357844 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.357822 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.381479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.381443 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl7kg\" (UniqueName: \"kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg\") pod \"maas-api-revoke-sub-3d4156f1-qhggr\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.475647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.475612 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:08.784544 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.784493 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr"] Oct 30 06:31:08.787465 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:31:08.787437 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb47ec55_40f3_45ae_b3a1_754002b94c19.slice/crio-cd7f949b521049a93c9366459ee813f606070a1f082cbee8ed42fbdf0dc6c655 WatchSource:0}: Error finding container cd7f949b521049a93c9366459ee813f606070a1f082cbee8ed42fbdf0dc6c655: Status 404 returned error can't find the container with id cd7f949b521049a93c9366459ee813f606070a1f082cbee8ed42fbdf0dc6c655 Oct 30 06:31:08.822577 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.822511 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:31:08.822751 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:08.822551 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:31:09.200611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:09.200560 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" event={"ID":"bb47ec55-40f3-45ae-b3a1-754002b94c19","Type":"ContainerStarted","Data":"8b80756cebfe82fedb3b5d818bb1ed6172f284a2bc1d34711366b76de0d65bde"} Oct 30 06:31:09.201171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:09.200633 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" event={"ID":"bb47ec55-40f3-45ae-b3a1-754002b94c19","Type":"ContainerStarted","Data":"cd7f949b521049a93c9366459ee813f606070a1f082cbee8ed42fbdf0dc6c655"} Oct 30 06:31:09.244914 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:09.244854 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" podStartSLOduration=1.244837956 podStartE2EDuration="1.244837956s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:31:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:31:09.243740727 +0000 UTC m=+3575.819427780" watchObservedRunningTime="2026-10-30 06:31:09.244837956 +0000 UTC m=+3575.820525009" Oct 30 06:31:09.566258 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:09.566222 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6238ddcb-2d47-4057-90f4-bf555049dcb9" path="/var/lib/kubelet/pods/6238ddcb-2d47-4057-90f4-bf555049dcb9/volumes" Oct 30 06:31:10.207647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:10.207609 5565 generic.go:356] "Generic (PLEG): container finished" podID="bb47ec55-40f3-45ae-b3a1-754002b94c19" containerID="8b80756cebfe82fedb3b5d818bb1ed6172f284a2bc1d34711366b76de0d65bde" exitCode=0 Oct 30 06:31:10.208059 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:10.207689 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" event={"ID":"bb47ec55-40f3-45ae-b3a1-754002b94c19","Type":"ContainerDied","Data":"8b80756cebfe82fedb3b5d818bb1ed6172f284a2bc1d34711366b76de0d65bde"} Oct 30 06:31:11.355443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.355416 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:11.492034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.491948 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg\" (UniqueName: \"kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg\") pod \"bb47ec55-40f3-45ae-b3a1-754002b94c19\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " Oct 30 06:31:11.492034 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.492003 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca\") pod \"bb47ec55-40f3-45ae-b3a1-754002b94c19\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " Oct 30 06:31:11.492290 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.492100 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca\") pod \"bb47ec55-40f3-45ae-b3a1-754002b94c19\" (UID: \"bb47ec55-40f3-45ae-b3a1-754002b94c19\") " Oct 30 06:31:11.492444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.492410 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca" pod "bb47ec55-40f3-45ae-b3a1-754002b94c19" (UID: "bb47ec55-40f3-45ae-b3a1-754002b94c19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:11.492588 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.492499 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:11.494059 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.494037 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca" pod "bb47ec55-40f3-45ae-b3a1-754002b94c19" (UID: "bb47ec55-40f3-45ae-b3a1-754002b94c19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:31:11.494194 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.494174 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg" pod "bb47ec55-40f3-45ae-b3a1-754002b94c19" (UID: "bb47ec55-40f3-45ae-b3a1-754002b94c19"). InnerVolumeSpecName "kube-api-access-tl7kg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:11.593352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.593322 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb47ec55-40f3-45ae-b3a1-754002b94c19-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:11.593352 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:11.593349 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl7kg\" (UniqueName: \"kubernetes.io/projected/bb47ec55-40f3-45ae-b3a1-754002b94c19-kube-api-access-tl7kg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:12.223847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:12.223813 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" Oct 30 06:31:12.223847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:12.223829 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr" event={"ID":"bb47ec55-40f3-45ae-b3a1-754002b94c19","Type":"ContainerDied","Data":"cd7f949b521049a93c9366459ee813f606070a1f082cbee8ed42fbdf0dc6c655"} Oct 30 06:31:12.224061 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:12.223880 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd7f949b521049a93c9366459ee813f606070a1f082cbee8ed42fbdf0dc6c655" Oct 30 06:31:17.531041 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:31:17.531003 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:31:18.822849 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:18.822811 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:31:18.823238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:18.822897 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:31:20.182625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:20.182587 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:31:22.315539 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.315318 5565 generic.go:356] "Generic (PLEG): container finished" podID="056f5b7e-883c-4898-ad98-a0834c18b154" containerID="401f3412b1a1d72732f2442eb6d53d9868fb3a7cbd442b5c202359038ab6f006" exitCode=0 Oct 30 06:31:22.315539 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.315406 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" event={"ID":"056f5b7e-883c-4898-ad98-a0834c18b154","Type":"ContainerDied","Data":"401f3412b1a1d72732f2442eb6d53d9868fb3a7cbd442b5c202359038ab6f006"} Oct 30 06:31:22.318227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.318094 5565 generic.go:356] "Generic (PLEG): container finished" podID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerID="92f15f9a517ecb2990ff04de5bf516e4ff9b4f254b11e627f42d6a2e0459b43e" exitCode=0 Oct 30 06:31:22.318227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.318162 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" event={"ID":"c04e21e0-afaa-4dd6-bf24-936d5a30438b","Type":"ContainerDied","Data":"92f15f9a517ecb2990ff04de5bf516e4ff9b4f254b11e627f42d6a2e0459b43e"} Oct 30 06:31:22.517947 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.517904 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:31:22.580562 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.580517 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:31:22.636863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.636831 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w\" (UniqueName: \"kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w\") pod \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " Oct 30 06:31:22.637032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.636872 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume\") pod \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\" (UID: \"c04e21e0-afaa-4dd6-bf24-936d5a30438b\") " Oct 30 06:31:22.637295 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.637273 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume" pod "c04e21e0-afaa-4dd6-bf24-936d5a30438b" (UID: "c04e21e0-afaa-4dd6-bf24-936d5a30438b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:22.639042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.639023 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w" pod "c04e21e0-afaa-4dd6-bf24-936d5a30438b" (UID: "c04e21e0-afaa-4dd6-bf24-936d5a30438b"). InnerVolumeSpecName "kube-api-access-wlh8w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:22.737552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.737487 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume\") pod \"056f5b7e-883c-4898-ad98-a0834c18b154\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " Oct 30 06:31:22.737733 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.737589 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c\" (UniqueName: \"kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c\") pod \"056f5b7e-883c-4898-ad98-a0834c18b154\" (UID: \"056f5b7e-883c-4898-ad98-a0834c18b154\") " Oct 30 06:31:22.737869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.737842 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume" pod "056f5b7e-883c-4898-ad98-a0834c18b154" (UID: "056f5b7e-883c-4898-ad98-a0834c18b154"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:31:22.738013 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.737997 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c04e21e0-afaa-4dd6-bf24-936d5a30438b-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:22.738088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.738018 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/056f5b7e-883c-4898-ad98-a0834c18b154-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:22.738088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.738032 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlh8w\" (UniqueName: \"kubernetes.io/projected/c04e21e0-afaa-4dd6-bf24-936d5a30438b-kube-api-access-wlh8w\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:22.739584 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.739564 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c" pod "056f5b7e-883c-4898-ad98-a0834c18b154" (UID: "056f5b7e-883c-4898-ad98-a0834c18b154"). InnerVolumeSpecName "kube-api-access-wl22c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:31:22.839457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:22.839370 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wl22c\" (UniqueName: \"kubernetes.io/projected/056f5b7e-883c-4898-ad98-a0834c18b154-kube-api-access-wl22c\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:31:23.325799 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.325744 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" event={"ID":"056f5b7e-883c-4898-ad98-a0834c18b154","Type":"ContainerDied","Data":"a1d7576579de95ba33e1c08e10bb3bebc48f531d79414e5988809c5122b1ff75"} Oct 30 06:31:23.326326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.325884 5565 scope.go:122] "RemoveContainer" containerID="401f3412b1a1d72732f2442eb6d53d9868fb3a7cbd442b5c202359038ab6f006" Oct 30 06:31:23.326326 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.326081 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp" Oct 30 06:31:23.334772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.334675 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" event={"ID":"c04e21e0-afaa-4dd6-bf24-936d5a30438b","Type":"ContainerDied","Data":"d57c82fddf06b5eb792e008d558c71c7fb55c9f985befe7fa9574a6a5ad4dbe4"} Oct 30 06:31:23.334772 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.334724 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g" Oct 30 06:31:23.343707 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.343689 5565 scope.go:122] "RemoveContainer" containerID="92f15f9a517ecb2990ff04de5bf516e4ff9b4f254b11e627f42d6a2e0459b43e" Oct 30 06:31:23.389265 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.389233 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g"] Oct 30 06:31:23.403174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.403151 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-627222-5546c6d54j7w2g"] Oct 30 06:31:23.430983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.430957 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp"] Oct 30 06:31:23.437819 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.437799 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-627222-c9cf9s86fp"] Oct 30 06:31:23.565892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.565851 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" path="/var/lib/kubelet/pods/056f5b7e-883c-4898-ad98-a0834c18b154/volumes" Oct 30 06:31:23.566323 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:23.566302 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" path="/var/lib/kubelet/pods/c04e21e0-afaa-4dd6-bf24-936d5a30438b/volumes" Oct 30 06:31:25.596502 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:31:25.596460 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:31:47.535685 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:31:47.535580 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:32:17.531874 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:32:17.531829 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:32:47.530889 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:32:47.530848 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:32:53.989656 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:53.989607 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:32:58.027361 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.027327 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65"] Oct 30 06:32:58.041120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.041095 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-hqb65"] Oct 30 06:32:58.381023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.380945 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9"] Oct 30 06:32:58.381336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381323 5565 cpu_manager.go:414] "containerMap: removing container" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" containerName="payload-pre-processing" Oct 30 06:32:58.381393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381336 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" containerName="payload-pre-processing" Oct 30 06:32:58.381393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381350 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerName="payload-processing" Oct 30 06:32:58.381393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381355 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerName="payload-processing" Oct 30 06:32:58.381393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381375 5565 cpu_manager.go:414] "containerMap: removing container" podUID="bb47ec55-40f3-45ae-b3a1-754002b94c19" containerName="revoke-keys" Oct 30 06:32:58.381393 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381380 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb47ec55-40f3-45ae-b3a1-754002b94c19" containerName="revoke-keys" Oct 30 06:32:58.381595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381440 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef401418-87c5-4ca7-bf26-64660c5a6545" containerName="cleanup" Oct 30 06:32:58.381595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381449 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="056f5b7e-883c-4898-ad98-a0834c18b154" containerName="payload-pre-processing" Oct 30 06:32:58.381595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381458 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c04e21e0-afaa-4dd6-bf24-936d5a30438b" containerName="payload-processing" Oct 30 06:32:58.381595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.381467 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb47ec55-40f3-45ae-b3a1-754002b94c19" containerName="revoke-keys" Oct 30 06:32:58.384738 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.384720 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.402933 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.402911 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:32:58.406658 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.406628 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9"] Oct 30 06:32:58.417041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.417017 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:32:58.417164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.417114 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:32:58.417238 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.417193 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:32:58.441372 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.441348 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvg7l\" (UniqueName: \"kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.441489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.441390 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.441550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.441483 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.542050 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.542013 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.542181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.542105 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.542181 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.542173 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvg7l\" (UniqueName: \"kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.542747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.542727 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.544296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.544272 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.566892 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.566867 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvg7l\" (UniqueName: \"kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l\") pod \"maas-api-revoke-sub-d576e68b-czpj9\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.697386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.697335 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:32:58.994012 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:58.993988 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9"] Oct 30 06:32:58.996399 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:32:58.996371 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd49e77_7489_4696_a6be_46882366d99d.slice/crio-9ad9058b34b854d843f0b3376deb969528ac584e756231e59aeb2a49a42b8cd8 WatchSource:0}: Error finding container 9ad9058b34b854d843f0b3376deb969528ac584e756231e59aeb2a49a42b8cd8: Status 404 returned error can't find the container with id 9ad9058b34b854d843f0b3376deb969528ac584e756231e59aeb2a49a42b8cd8 Oct 30 06:32:59.565755 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:59.565684 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a032508f-628a-4955-b560-ee0bee711a7c" path="/var/lib/kubelet/pods/a032508f-628a-4955-b560-ee0bee711a7c/volumes" Oct 30 06:32:59.845260 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:59.845196 5565 generic.go:356] "Generic (PLEG): container finished" podID="7bd49e77-7489-4696-a6be-46882366d99d" containerID="64d0dcc6b4138a1d8c6db11dcfd53952dc132ba3a83095d0f46a903a54ffd843" exitCode=0 Oct 30 06:32:59.845374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:59.845258 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" event={"ID":"7bd49e77-7489-4696-a6be-46882366d99d","Type":"ContainerDied","Data":"64d0dcc6b4138a1d8c6db11dcfd53952dc132ba3a83095d0f46a903a54ffd843"} Oct 30 06:32:59.845374 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:32:59.845299 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" event={"ID":"7bd49e77-7489-4696-a6be-46882366d99d","Type":"ContainerStarted","Data":"9ad9058b34b854d843f0b3376deb969528ac584e756231e59aeb2a49a42b8cd8"} Oct 30 06:33:00.978019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:00.977993 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:33:01.061011 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.060982 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca\") pod \"7bd49e77-7489-4696-a6be-46882366d99d\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " Oct 30 06:33:01.061157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.061018 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l\" (UniqueName: \"kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l\") pod \"7bd49e77-7489-4696-a6be-46882366d99d\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " Oct 30 06:33:01.061157 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.061084 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca\") pod \"7bd49e77-7489-4696-a6be-46882366d99d\" (UID: \"7bd49e77-7489-4696-a6be-46882366d99d\") " Oct 30 06:33:01.061493 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.061468 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca" pod "7bd49e77-7489-4696-a6be-46882366d99d" (UID: "7bd49e77-7489-4696-a6be-46882366d99d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:01.063051 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.063026 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca" pod "7bd49e77-7489-4696-a6be-46882366d99d" (UID: "7bd49e77-7489-4696-a6be-46882366d99d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:01.063144 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.063120 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l" pod "7bd49e77-7489-4696-a6be-46882366d99d" (UID: "7bd49e77-7489-4696-a6be-46882366d99d"). InnerVolumeSpecName "kube-api-access-rvg7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:01.162442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.162387 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:33:01.162442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.162410 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvg7l\" (UniqueName: \"kubernetes.io/projected/7bd49e77-7489-4696-a6be-46882366d99d-kube-api-access-rvg7l\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:33:01.162442 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.162419 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7bd49e77-7489-4696-a6be-46882366d99d-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:33:01.855777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.855711 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" Oct 30 06:33:01.855925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.855710 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9" event={"ID":"7bd49e77-7489-4696-a6be-46882366d99d","Type":"ContainerDied","Data":"9ad9058b34b854d843f0b3376deb969528ac584e756231e59aeb2a49a42b8cd8"} Oct 30 06:33:01.855925 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:01.855835 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ad9058b34b854d843f0b3376deb969528ac584e756231e59aeb2a49a42b8cd8" Oct 30 06:33:09.554645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:09.554608 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:33:15.324750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.324670 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr"] Oct 30 06:33:15.327053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.325080 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7bd49e77-7489-4696-a6be-46882366d99d" containerName="revoke-keys" Oct 30 06:33:15.327053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.325092 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd49e77-7489-4696-a6be-46882366d99d" containerName="revoke-keys" Oct 30 06:33:15.327053 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.325191 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd49e77-7489-4696-a6be-46882366d99d" containerName="revoke-keys" Oct 30 06:33:15.328170 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.328144 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.349319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.349285 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr"] Oct 30 06:33:15.361266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.361227 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:33:15.361266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.361254 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:33:15.361510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.361270 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:33:15.361510 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.361292 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:33:15.444609 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.444574 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.444766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.444628 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvfzz\" (UniqueName: \"kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.444766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.444748 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.545341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.545314 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.545436 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.545370 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvfzz\" (UniqueName: \"kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.545476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.545441 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.546115 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.546090 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.547572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.547551 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.570268 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.570244 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvfzz\" (UniqueName: \"kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz\") pod \"maas-api-revoke-sub-d2bc93bc-4hmbr\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.641061 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.641002 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:15.942549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:15.942514 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr"] Oct 30 06:33:15.944975 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:33:15.944943 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6526d6e7_6777_4d47_abce_81f48bb08a4f.slice/crio-12595c3b26f2fb47c7371363d9fdba896e470431b099bad0603704f549aac8ab WatchSource:0}: Error finding container 12595c3b26f2fb47c7371363d9fdba896e470431b099bad0603704f549aac8ab: Status 404 returned error can't find the container with id 12595c3b26f2fb47c7371363d9fdba896e470431b099bad0603704f549aac8ab Oct 30 06:33:16.940647 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:16.940621 5565 generic.go:356] "Generic (PLEG): container finished" podID="6526d6e7-6777-4d47-abce-81f48bb08a4f" containerID="b5bcfa4d3af6b142726d3babd88f5047777eca56cfd272673833ad58045d5430" exitCode=0 Oct 30 06:33:16.940988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:16.940714 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" event={"ID":"6526d6e7-6777-4d47-abce-81f48bb08a4f","Type":"ContainerDied","Data":"b5bcfa4d3af6b142726d3babd88f5047777eca56cfd272673833ad58045d5430"} Oct 30 06:33:16.940988 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:16.940763 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" event={"ID":"6526d6e7-6777-4d47-abce-81f48bb08a4f","Type":"ContainerStarted","Data":"12595c3b26f2fb47c7371363d9fdba896e470431b099bad0603704f549aac8ab"} Oct 30 06:33:17.531907 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:33:17.531871 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:33:18.027795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.027763 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5"] Oct 30 06:33:18.042545 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.042489 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-5jxv5"] Oct 30 06:33:18.080896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.080874 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:18.167365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.167338 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca\") pod \"6526d6e7-6777-4d47-abce-81f48bb08a4f\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " Oct 30 06:33:18.167573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.167438 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca\") pod \"6526d6e7-6777-4d47-abce-81f48bb08a4f\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " Oct 30 06:33:18.167573 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.167497 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz\" (UniqueName: \"kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz\") pod \"6526d6e7-6777-4d47-abce-81f48bb08a4f\" (UID: \"6526d6e7-6777-4d47-abce-81f48bb08a4f\") " Oct 30 06:33:18.167728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.167701 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca" pod "6526d6e7-6777-4d47-abce-81f48bb08a4f" (UID: "6526d6e7-6777-4d47-abce-81f48bb08a4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:33:18.167840 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.167823 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:33:18.169398 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.169374 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca" pod "6526d6e7-6777-4d47-abce-81f48bb08a4f" (UID: "6526d6e7-6777-4d47-abce-81f48bb08a4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:33:18.169490 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.169398 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz" pod "6526d6e7-6777-4d47-abce-81f48bb08a4f" (UID: "6526d6e7-6777-4d47-abce-81f48bb08a4f"). InnerVolumeSpecName "kube-api-access-lvfzz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:33:18.269114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.269075 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6526d6e7-6777-4d47-abce-81f48bb08a4f-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:33:18.269114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.269106 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvfzz\" (UniqueName: \"kubernetes.io/projected/6526d6e7-6777-4d47-abce-81f48bb08a4f-kube-api-access-lvfzz\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:33:18.955324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.955282 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" event={"ID":"6526d6e7-6777-4d47-abce-81f48bb08a4f","Type":"ContainerDied","Data":"12595c3b26f2fb47c7371363d9fdba896e470431b099bad0603704f549aac8ab"} Oct 30 06:33:18.955324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.955309 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr" Oct 30 06:33:18.955572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:18.955335 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12595c3b26f2fb47c7371363d9fdba896e470431b099bad0603704f549aac8ab" Oct 30 06:33:19.566880 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:19.566841 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e221f9e-8995-466c-8f2c-829b3eb3ae58" path="/var/lib/kubelet/pods/0e221f9e-8995-466c-8f2c-829b3eb3ae58/volumes" Oct 30 06:33:32.660692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.660661 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v"] Oct 30 06:33:32.661152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.661083 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6526d6e7-6777-4d47-abce-81f48bb08a4f" containerName="revoke-keys" Oct 30 06:33:32.661152 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.661096 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6526d6e7-6777-4d47-abce-81f48bb08a4f" containerName="revoke-keys" Oct 30 06:33:32.661228 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.661176 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6526d6e7-6777-4d47-abce-81f48bb08a4f" containerName="revoke-keys" Oct 30 06:33:32.665816 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.665793 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.682817 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.682796 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:33:32.697017 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.696991 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:33:32.697169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.696996 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wbd8\"" Oct 30 06:33:32.697169 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.697059 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:33:32.698878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.698855 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v"] Oct 30 06:33:32.699478 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.699453 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p8ncn\" (UniqueName: \"kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.699572 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.699500 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.699644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.699617 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.699721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.699696 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.704164 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.704139 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-be81b5-serving-cert\"" Oct 30 06:33:32.717446 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.717420 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds"] Oct 30 06:33:32.722264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.722239 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.740834 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.740806 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds"] Oct 30 06:33:32.766760 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.766734 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm"] Oct 30 06:33:32.771226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.771204 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:32.782920 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.782901 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-be81b5\"" Oct 30 06:33:32.789641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.789617 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm"] Oct 30 06:33:32.791175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.791143 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-0e176d0a-dockercfg-5nkqs\"" Oct 30 06:33:32.800869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.800842 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.800970 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.800946 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.801020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801000 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:32.801130 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801111 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fthm\" (UniqueName: \"kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm\") pod \"payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:32.801173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801147 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l76fv\" (UniqueName: \"kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv\") pod \"payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.801229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801194 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p8ncn\" (UniqueName: \"kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.801371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801350 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.801444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801387 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.801444 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.801431 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.801565 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:33:32.801481 5565 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-serving-cert: secret "maas-api-e2e-worker-single-main-be81b5-serving-cert" not found Oct 30 06:33:32.801565 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:33:32.801556 5565 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls podName:c974d984-16b4-48dc-96f1-45cb5c5fd11e nodeName:}" failed. No retries permitted until 2026-10-30 06:33:33.301540341 +0000 UTC m=+3719.877227381 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls") pod "maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" (UID: "c974d984-16b4-48dc-96f1-45cb5c5fd11e") : secret "maas-api-e2e-worker-single-main-be81b5-serving-cert" not found Oct 30 06:33:32.803753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.803729 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.826990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.826967 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p8ncn\" (UniqueName: \"kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:32.902581 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.902547 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fthm\" (UniqueName: \"kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm\") pod \"payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:32.902775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.902592 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l76fv\" (UniqueName: \"kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv\") pod \"payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.902775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.902710 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.902775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.902742 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:32.903462 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.903435 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.903597 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.903501 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:32.928390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.928325 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l76fv\" (UniqueName: \"kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv\") pod \"payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:32.928390 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:32.928325 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fthm\" (UniqueName: \"kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm\") pod \"payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:33.034652 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.034618 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:33.084280 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.084219 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:33.306596 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.306562 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:33.309368 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.309341 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:33.335489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.335468 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds"] Oct 30 06:33:33.337767 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:33:33.337742 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f02c14c_b280_4d38_816c_964fb7df383e.slice/crio-36e2dd416cc6ca057a233fc09dc6723bbecd4e98dc5f51bffb24496fb1902396 WatchSource:0}: Error finding container 36e2dd416cc6ca057a233fc09dc6723bbecd4e98dc5f51bffb24496fb1902396: Status 404 returned error can't find the container with id 36e2dd416cc6ca057a233fc09dc6723bbecd4e98dc5f51bffb24496fb1902396 Oct 30 06:33:33.391691 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.391663 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm"] Oct 30 06:33:33.392964 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:33:33.392940 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6260496b_4770_4dae_b43c_e280d236ef18.slice/crio-486f7446e4a3061b55279188ca8e37b230c53cfc7ee1da88a4c8905c1def7373 WatchSource:0}: Error finding container 486f7446e4a3061b55279188ca8e37b230c53cfc7ee1da88a4c8905c1def7373: Status 404 returned error can't find the container with id 486f7446e4a3061b55279188ca8e37b230c53cfc7ee1da88a4c8905c1def7373 Oct 30 06:33:33.579515 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.579438 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:33.879102 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:33.879079 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v"] Oct 30 06:33:33.881575 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:33:33.881549 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc974d984_16b4_48dc_96f1_45cb5c5fd11e.slice/crio-ba03cb6a5e99325c20715e48437f9c05a87d1e3ab43f8caa40510c49c07b5c71 WatchSource:0}: Error finding container ba03cb6a5e99325c20715e48437f9c05a87d1e3ab43f8caa40510c49c07b5c71: Status 404 returned error can't find the container with id ba03cb6a5e99325c20715e48437f9c05a87d1e3ab43f8caa40510c49c07b5c71 Oct 30 06:33:34.026600 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.026556 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" event={"ID":"c974d984-16b4-48dc-96f1-45cb5c5fd11e","Type":"ContainerStarted","Data":"ba03cb6a5e99325c20715e48437f9c05a87d1e3ab43f8caa40510c49c07b5c71"} Oct 30 06:33:34.028076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.028041 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" event={"ID":"4f02c14c-b280-4d38-816c-964fb7df383e","Type":"ContainerStarted","Data":"29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8"} Oct 30 06:33:34.028226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.028099 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" event={"ID":"4f02c14c-b280-4d38-816c-964fb7df383e","Type":"ContainerStarted","Data":"36e2dd416cc6ca057a233fc09dc6723bbecd4e98dc5f51bffb24496fb1902396"} Oct 30 06:33:34.028226 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.028129 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:34.029550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.029503 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" event={"ID":"6260496b-4770-4dae-b43c-e280d236ef18","Type":"ContainerStarted","Data":"3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e"} Oct 30 06:33:34.029679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.029554 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" event={"ID":"6260496b-4770-4dae-b43c-e280d236ef18","Type":"ContainerStarted","Data":"486f7446e4a3061b55279188ca8e37b230c53cfc7ee1da88a4c8905c1def7373"} Oct 30 06:33:34.029679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.029639 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:34.086853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.086794 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" podStartSLOduration=2.086781321 podStartE2EDuration="2.086781321s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:34.072586627 +0000 UTC m=+3720.648273674" watchObservedRunningTime="2026-10-30 06:33:34.086781321 +0000 UTC m=+3720.662468374" Oct 30 06:33:34.122644 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:34.122604 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" podStartSLOduration=2.122591702 podStartE2EDuration="2.122591702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:34.121456184 +0000 UTC m=+3720.697143238" watchObservedRunningTime="2026-10-30 06:33:34.122591702 +0000 UTC m=+3720.698278791" Oct 30 06:33:35.035750 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:35.035710 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" event={"ID":"c974d984-16b4-48dc-96f1-45cb5c5fd11e","Type":"ContainerStarted","Data":"9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0"} Oct 30 06:33:35.036161 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:35.036019 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:35.082958 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:35.082911 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" podStartSLOduration=2.708112978 podStartE2EDuration="3.082898435s" totalImagesPullingTime="374.785457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:33:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:33:35.08150535 +0000 UTC m=+3721.657192403" watchObservedRunningTime="2026-10-30 06:33:35.082898435 +0000 UTC m=+3721.658585488" Oct 30 06:33:36.464618 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:36.464577 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:33:39.027793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:39.027759 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps"] Oct 30 06:33:39.041578 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:39.041553 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-69vps"] Oct 30 06:33:39.565032 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:39.565001 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1cd4b85-f6f1-4c16-a86f-5d787e86fd95" path="/var/lib/kubelet/pods/c1cd4b85-f6f1-4c16-a86f-5d787e86fd95/volumes" Oct 30 06:33:41.045729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:41.045701 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:33:45.039308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:45.039273 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:33:45.039825 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:45.039344 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:33:47.531872 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:33:47.531835 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:33:50.782112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:33:50.782078 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:34:05.481210 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:05.481173 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:34:17.530885 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:34:17.530846 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:34:18.790775 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:18.789515 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:34:28.870290 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:28.870254 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds"] Oct 30 06:34:28.889457 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:28.889428 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm"] Oct 30 06:34:29.664521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:29.664490 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v"] Oct 30 06:34:29.664719 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:29.664685 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" podUID="c974d984-16b4-48dc-96f1-45cb5c5fd11e" containerName="maas-api" containerID="cri-o://9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0" gracePeriod=30 Oct 30 06:34:30.112227 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.112199 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:34:30.239371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.239304 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\") pod \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " Oct 30 06:34:30.239371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.239361 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn\" (UniqueName: \"kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn\") pod \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " Oct 30 06:34:30.239634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.239390 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca\") pod \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " Oct 30 06:34:30.239634 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.239441 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs\") pod \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\" (UID: \"c974d984-16b4-48dc-96f1-45cb5c5fd11e\") " Oct 30 06:34:30.239866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.239835 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca" pod "c974d984-16b4-48dc-96f1-45cb5c5fd11e" (UID: "c974d984-16b4-48dc-96f1-45cb5c5fd11e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:34:30.241627 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.241597 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls" pod "c974d984-16b4-48dc-96f1-45cb5c5fd11e" (UID: "c974d984-16b4-48dc-96f1-45cb5c5fd11e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:34:30.241747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.241636 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn" pod "c974d984-16b4-48dc-96f1-45cb5c5fd11e" (UID: "c974d984-16b4-48dc-96f1-45cb5c5fd11e"). InnerVolumeSpecName "kube-api-access-p8ncn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:34:30.241747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.241663 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs" pod "c974d984-16b4-48dc-96f1-45cb5c5fd11e" (UID: "c974d984-16b4-48dc-96f1-45cb5c5fd11e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:34:30.320997 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.320969 5565 generic.go:356] "Generic (PLEG): container finished" podID="c974d984-16b4-48dc-96f1-45cb5c5fd11e" containerID="9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0" exitCode=0 Oct 30 06:34:30.321114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.321018 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" Oct 30 06:34:30.321114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.321031 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" event={"ID":"c974d984-16b4-48dc-96f1-45cb5c5fd11e","Type":"ContainerDied","Data":"9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0"} Oct 30 06:34:30.321114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.321084 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v" event={"ID":"c974d984-16b4-48dc-96f1-45cb5c5fd11e","Type":"ContainerDied","Data":"ba03cb6a5e99325c20715e48437f9c05a87d1e3ab43f8caa40510c49c07b5c71"} Oct 30 06:34:30.321236 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.321114 5565 scope.go:122] "RemoveContainer" containerID="9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0" Oct 30 06:34:30.330741 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.330723 5565 scope.go:122] "RemoveContainer" containerID="9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0" Oct 30 06:34:30.331033 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:34:30.331012 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0\": container with ID starting with 9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0 not found: ID does not exist" containerID="9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0" Oct 30 06:34:30.331090 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.331041 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0"} err="failed to get container status \"9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0\": rpc error: code = NotFound desc = could not find container \"9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0\": container with ID starting with 9632939fca7b31e5822feb795db537f368844434e481285d7b258a2a66e3f9a0 not found: ID does not exist" Oct 30 06:34:30.341148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.341122 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p8ncn\" (UniqueName: \"kubernetes.io/projected/c974d984-16b4-48dc-96f1-45cb5c5fd11e-kube-api-access-p8ncn\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:30.341148 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.341145 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c974d984-16b4-48dc-96f1-45cb5c5fd11e-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:30.341279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.341156 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:30.341279 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.341165 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c974d984-16b4-48dc-96f1-45cb5c5fd11e-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:30.378495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.378470 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v"] Oct 30 06:34:30.385129 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:30.385096 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-be81b5-6f8c5bb97d-v7q8v"] Oct 30 06:34:31.565896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:31.565865 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c974d984-16b4-48dc-96f1-45cb5c5fd11e" path="/var/lib/kubelet/pods/c974d984-16b4-48dc-96f1-45cb5c5fd11e/volumes" Oct 30 06:34:31.996305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:31.996216 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:34:33.871311 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:33.871274 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" containerName="payload-processing" containerID="cri-o://29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8" gracePeriod=25 Oct 30 06:34:33.889724 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:33.889680 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" podUID="6260496b-4770-4dae-b43c-e280d236ef18" containerName="payload-pre-processing" containerID="cri-o://3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e" gracePeriod=25 Oct 30 06:34:35.037731 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:35.037692 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:34:35.038174 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:35.037694 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" podUID="6260496b-4770-4dae-b43c-e280d236ef18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:34:40.057060 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:40.057026 5565 scope.go:122] "RemoveContainer" containerID="02707d8bbc682dbc344deadfac61c7f245e11ede43246c24fee396b84496f280" Oct 30 06:34:40.065593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:40.065569 5565 scope.go:122] "RemoveContainer" containerID="d17db91fd341c96bd0390b387e7708afd6fce49b9dae7ef5f9c39c17efa0760a" Oct 30 06:34:40.073211 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:40.073187 5565 scope.go:122] "RemoveContainer" containerID="c1d70356bffb4d38c88fb2d310bd2c0c528a5e1ec1ee3a0ec7e0689330a5e67d" Oct 30 06:34:42.382965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.382921 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh"] Oct 30 06:34:42.383505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.383319 5565 cpu_manager.go:414] "containerMap: removing container" podUID="c974d984-16b4-48dc-96f1-45cb5c5fd11e" containerName="maas-api" Oct 30 06:34:42.383505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.383330 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c974d984-16b4-48dc-96f1-45cb5c5fd11e" containerName="maas-api" Oct 30 06:34:42.383505 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.383419 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="c974d984-16b4-48dc-96f1-45cb5c5fd11e" containerName="maas-api" Oct 30 06:34:42.388025 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.387986 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.420648 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.420625 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-b658b769-kserve-self-signed-certs\"" Oct 30 06:34:42.422351 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.422325 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh"] Oct 30 06:34:42.545460 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.545431 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txpzl\" (UniqueName: \"kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.545595 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.545471 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.545651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.545603 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.545651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.545636 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.545736 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.545717 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.545774 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.545756 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647090 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647175 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647143 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647183 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647207 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647242 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txpzl\" (UniqueName: \"kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647423 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647353 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647565 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647711 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647654 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.647792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.647723 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.649603 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.649574 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.649887 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.649864 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.671742 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.671720 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txpzl\" (UniqueName: \"kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl\") pod \"e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.700023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.699998 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:42.997677 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:42.997444 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh"] Oct 30 06:34:43.000163 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:34:43.000132 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e165a8c_0f60_49e6_ac8b_5f9912812892.slice/crio-a91d503f0258c2c4a061a8492a5df4ed4f1dda0d3c168c6c4b38e2497bcf55e7 WatchSource:0}: Error finding container a91d503f0258c2c4a061a8492a5df4ed4f1dda0d3c168c6c4b38e2497bcf55e7: Status 404 returned error can't find the container with id a91d503f0258c2c4a061a8492a5df4ed4f1dda0d3c168c6c4b38e2497bcf55e7 Oct 30 06:34:43.396709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:43.396669 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" event={"ID":"3e165a8c-0f60-49e6-ac8b-5f9912812892","Type":"ContainerStarted","Data":"0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087"} Oct 30 06:34:43.396709 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:43.396710 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" event={"ID":"3e165a8c-0f60-49e6-ac8b-5f9912812892","Type":"ContainerStarted","Data":"a91d503f0258c2c4a061a8492a5df4ed4f1dda0d3c168c6c4b38e2497bcf55e7"} Oct 30 06:34:43.442749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:43.442702 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" podStartSLOduration=1.4426876260000001 podStartE2EDuration="1.442687626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:34:43.441397503 +0000 UTC m=+3790.017084558" watchObservedRunningTime="2026-10-30 06:34:43.442687626 +0000 UTC m=+3790.018374678" Oct 30 06:34:43.700288 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:43.700208 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:44.400812 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:44.400777 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:34:45.056552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:45.054657 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:34:45.056552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:45.055302 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" podUID="6260496b-4770-4dae-b43c-e280d236ef18" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 06:34:47.638726 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:34:47.638689 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:34:52.700242 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:52.700210 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:52.712336 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:52.712315 5565 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:53.475023 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:53.474990 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:34:54.142767 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.142708 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:34:54.166443 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.166425 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:34:54.181560 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.181516 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume\") pod \"6260496b-4770-4dae-b43c-e280d236ef18\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " Oct 30 06:34:54.181659 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.181586 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume\") pod \"4f02c14c-b280-4d38-816c-964fb7df383e\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " Oct 30 06:34:54.181705 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.181696 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm\" (UniqueName: \"kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm\") pod \"6260496b-4770-4dae-b43c-e280d236ef18\" (UID: \"6260496b-4770-4dae-b43c-e280d236ef18\") " Oct 30 06:34:54.181742 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.181718 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv\" (UniqueName: \"kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv\") pod \"4f02c14c-b280-4d38-816c-964fb7df383e\" (UID: \"4f02c14c-b280-4d38-816c-964fb7df383e\") " Oct 30 06:34:54.181870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.181846 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume" pod "6260496b-4770-4dae-b43c-e280d236ef18" (UID: "6260496b-4770-4dae-b43c-e280d236ef18"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:34:54.181917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.181900 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume" pod "4f02c14c-b280-4d38-816c-964fb7df383e" (UID: "4f02c14c-b280-4d38-816c-964fb7df383e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:34:54.182028 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.182014 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4f02c14c-b280-4d38-816c-964fb7df383e-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:54.182067 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.182031 5565 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6260496b-4770-4dae-b43c-e280d236ef18-plugins-config-volume\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:54.183663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.183641 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv" pod "4f02c14c-b280-4d38-816c-964fb7df383e" (UID: "4f02c14c-b280-4d38-816c-964fb7df383e"). InnerVolumeSpecName "kube-api-access-l76fv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:34:54.183728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.183693 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm" pod "6260496b-4770-4dae-b43c-e280d236ef18" (UID: "6260496b-4770-4dae-b43c-e280d236ef18"). InnerVolumeSpecName "kube-api-access-8fthm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:34:54.282520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.282491 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fthm\" (UniqueName: \"kubernetes.io/projected/6260496b-4770-4dae-b43c-e280d236ef18-kube-api-access-8fthm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:54.282520 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.282517 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l76fv\" (UniqueName: \"kubernetes.io/projected/4f02c14c-b280-4d38-816c-964fb7df383e-kube-api-access-l76fv\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:34:54.469663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.469587 5565 generic.go:356] "Generic (PLEG): container finished" podID="4f02c14c-b280-4d38-816c-964fb7df383e" containerID="29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8" exitCode=0 Oct 30 06:34:54.469663 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.469634 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" Oct 30 06:34:54.469894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.469668 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" event={"ID":"4f02c14c-b280-4d38-816c-964fb7df383e","Type":"ContainerDied","Data":"29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8"} Oct 30 06:34:54.469894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.469725 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds" event={"ID":"4f02c14c-b280-4d38-816c-964fb7df383e","Type":"ContainerDied","Data":"36e2dd416cc6ca057a233fc09dc6723bbecd4e98dc5f51bffb24496fb1902396"} Oct 30 06:34:54.469894 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.469752 5565 scope.go:122] "RemoveContainer" containerID="29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8" Oct 30 06:34:54.471114 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.471093 5565 generic.go:356] "Generic (PLEG): container finished" podID="6260496b-4770-4dae-b43c-e280d236ef18" containerID="3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e" exitCode=0 Oct 30 06:34:54.471237 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.471121 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" event={"ID":"6260496b-4770-4dae-b43c-e280d236ef18","Type":"ContainerDied","Data":"3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e"} Oct 30 06:34:54.471237 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.471141 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" Oct 30 06:34:54.471237 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.471179 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm" event={"ID":"6260496b-4770-4dae-b43c-e280d236ef18","Type":"ContainerDied","Data":"486f7446e4a3061b55279188ca8e37b230c53cfc7ee1da88a4c8905c1def7373"} Oct 30 06:34:54.480815 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.480785 5565 scope.go:122] "RemoveContainer" containerID="29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8" Oct 30 06:34:54.481101 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:34:54.481083 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8\": container with ID starting with 29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8 not found: ID does not exist" containerID="29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8" Oct 30 06:34:54.481173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.481108 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8"} err="failed to get container status \"29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8\": rpc error: code = NotFound desc = could not find container \"29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8\": container with ID starting with 29f59079f6b890eb9b8cadb6f5681acc592df6d4fe178dd72a88f15f107f59b8 not found: ID does not exist" Oct 30 06:34:54.481173 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.481128 5565 scope.go:122] "RemoveContainer" containerID="3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e" Oct 30 06:34:54.489002 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.488984 5565 scope.go:122] "RemoveContainer" containerID="3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e" Oct 30 06:34:54.489272 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:34:54.489256 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e\": container with ID starting with 3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e not found: ID does not exist" containerID="3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e" Oct 30 06:34:54.489324 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.489277 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e"} err="failed to get container status \"3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e\": rpc error: code = NotFound desc = could not find container \"3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e\": container with ID starting with 3c842c3e86076724bc7c7603dc5faf8a708b665c4ef498fa55fc0045cec8b80e not found: ID does not exist" Oct 30 06:34:54.525625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.525603 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds"] Oct 30 06:34:54.539347 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.539325 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-be81b5-7db98d597thkds"] Oct 30 06:34:54.566795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.566774 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm"] Oct 30 06:34:54.570491 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:54.570471 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-be81b5-7f465rz6xm"] Oct 30 06:34:55.565613 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:55.565581 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" path="/var/lib/kubelet/pods/4f02c14c-b280-4d38-816c-964fb7df383e/volumes" Oct 30 06:34:55.565991 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:55.565953 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6260496b-4770-4dae-b43c-e280d236ef18" path="/var/lib/kubelet/pods/6260496b-4770-4dae-b43c-e280d236ef18/volumes" Oct 30 06:34:59.717805 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:34:59.717767 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:35:15.849582 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:15.849480 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:35:17.147411 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.147373 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w"] Oct 30 06:35:17.148037 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.148015 5565 cpu_manager.go:414] "containerMap: removing container" podUID="6260496b-4770-4dae-b43c-e280d236ef18" containerName="payload-pre-processing" Oct 30 06:35:17.148088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.148042 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6260496b-4770-4dae-b43c-e280d236ef18" containerName="payload-pre-processing" Oct 30 06:35:17.148088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.148081 5565 cpu_manager.go:414] "containerMap: removing container" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" containerName="payload-processing" Oct 30 06:35:17.148168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.148091 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" containerName="payload-processing" Oct 30 06:35:17.148205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.148189 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="6260496b-4770-4dae-b43c-e280d236ef18" containerName="payload-pre-processing" Oct 30 06:35:17.148240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.148210 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f02c14c-b280-4d38-816c-964fb7df383e" containerName="payload-processing" Oct 30 06:35:17.151932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.151913 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.168749 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.168720 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:35:17.172878 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.172849 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w"] Oct 30 06:35:17.182866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.182831 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:35:17.182866 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.182831 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:35:17.183074 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.182915 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:35:17.246089 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.246048 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.246262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.246112 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nkgg\" (UniqueName: \"kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.246262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.246247 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.347120 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.347075 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.347306 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.347169 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.347306 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.347223 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nkgg\" (UniqueName: \"kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.347990 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.347961 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.349665 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.349638 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.372740 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.372707 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nkgg\" (UniqueName: \"kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg\") pod \"maas-api-revoke-sub-452b6d7b-zvd7w\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.465036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.464945 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:17.670111 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:35:17.669829 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:35:17.946870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:17.946844 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w"] Oct 30 06:35:17.949239 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:35:17.949203 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d6712b1_9033_465b_b9c1_e45f393de1ce.slice/crio-d813fd36b3463b05cf2ed202acde4da2711d094fda11b0fb327156a204398f05 WatchSource:0}: Error finding container d813fd36b3463b05cf2ed202acde4da2711d094fda11b0fb327156a204398f05: Status 404 returned error can't find the container with id d813fd36b3463b05cf2ed202acde4da2711d094fda11b0fb327156a204398f05 Oct 30 06:35:18.635668 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:18.635627 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" event={"ID":"7d6712b1-9033-465b-b9c1-e45f393de1ce","Type":"ContainerStarted","Data":"a3c471365051de597be40537d2001196053f2774c4a0aa0a0e56d7743d8fc01f"} Oct 30 06:35:18.636052 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:18.635690 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" event={"ID":"7d6712b1-9033-465b-b9c1-e45f393de1ce","Type":"ContainerStarted","Data":"d813fd36b3463b05cf2ed202acde4da2711d094fda11b0fb327156a204398f05"} Oct 30 06:35:18.686277 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:18.686162 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" podStartSLOduration=1.6861434690000001 podStartE2EDuration="1.686143469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:35:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:35:18.685061399 +0000 UTC m=+3825.260748446" watchObservedRunningTime="2026-10-30 06:35:18.686143469 +0000 UTC m=+3825.261830522" Oct 30 06:35:19.642303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:19.642271 5565 generic.go:356] "Generic (PLEG): container finished" podID="7d6712b1-9033-465b-b9c1-e45f393de1ce" containerID="a3c471365051de597be40537d2001196053f2774c4a0aa0a0e56d7743d8fc01f" exitCode=0 Oct 30 06:35:19.642728 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:19.642355 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" event={"ID":"7d6712b1-9033-465b-b9c1-e45f393de1ce","Type":"ContainerDied","Data":"a3c471365051de597be40537d2001196053f2774c4a0aa0a0e56d7743d8fc01f"} Oct 30 06:35:20.777375 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.777344 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:20.834782 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.834752 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca\") pod \"7d6712b1-9033-465b-b9c1-e45f393de1ce\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " Oct 30 06:35:20.834923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.834802 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca\") pod \"7d6712b1-9033-465b-b9c1-e45f393de1ce\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " Oct 30 06:35:20.834923 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.834876 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg\" (UniqueName: \"kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg\") pod \"7d6712b1-9033-465b-b9c1-e45f393de1ce\" (UID: \"7d6712b1-9033-465b-b9c1-e45f393de1ce\") " Oct 30 06:35:20.835112 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.835091 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca" pod "7d6712b1-9033-465b-b9c1-e45f393de1ce" (UID: "7d6712b1-9033-465b-b9c1-e45f393de1ce"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:20.835244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.835227 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:20.836818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.836795 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca" pod "7d6712b1-9033-465b-b9c1-e45f393de1ce" (UID: "7d6712b1-9033-465b-b9c1-e45f393de1ce"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:20.836984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.836971 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg" pod "7d6712b1-9033-465b-b9c1-e45f393de1ce" (UID: "7d6712b1-9033-465b-b9c1-e45f393de1ce"). InnerVolumeSpecName "kube-api-access-6nkgg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:20.935875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.935815 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d6712b1-9033-465b-b9c1-e45f393de1ce-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:20.935875 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:20.935842 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nkgg\" (UniqueName: \"kubernetes.io/projected/7d6712b1-9033-465b-b9c1-e45f393de1ce-kube-api-access-6nkgg\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:21.652151 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:21.652121 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" Oct 30 06:35:21.652303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:21.652162 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-zvd7w" event={"ID":"7d6712b1-9033-465b-b9c1-e45f393de1ce","Type":"ContainerDied","Data":"d813fd36b3463b05cf2ed202acde4da2711d094fda11b0fb327156a204398f05"} Oct 30 06:35:21.652303 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:21.652214 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d813fd36b3463b05cf2ed202acde4da2711d094fda11b0fb327156a204398f05" Oct 30 06:35:28.369378 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:28.369338 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh"] Oct 30 06:35:30.083611 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:30.083576 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:35:34.814552 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.814426 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt"] Oct 30 06:35:34.818550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.815132 5565 cpu_manager.go:414] "containerMap: removing container" podUID="7d6712b1-9033-465b-b9c1-e45f393de1ce" containerName="revoke-keys" Oct 30 06:35:34.818550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.815158 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d6712b1-9033-465b-b9c1-e45f393de1ce" containerName="revoke-keys" Oct 30 06:35:34.818550 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.815269 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d6712b1-9033-465b-b9c1-e45f393de1ce" containerName="revoke-keys" Oct 30 06:35:34.835786 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.835749 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt"] Oct 30 06:35:34.835998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.835871 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:34.873548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.868860 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:35:34.873548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.868988 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:35:34.873548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.869080 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:35:34.873548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.869123 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:35:34.897551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.888158 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:34.897551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.888286 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:34.913551 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.911887 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m95lw\" (UniqueName: \"kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:34.935977 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:34.935946 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:35:35.017096 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.017052 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.033641 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.033607 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.033965 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.033944 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m95lw\" (UniqueName: \"kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.034642 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.032800 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.035475 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.035452 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.094323 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.094257 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m95lw\" (UniqueName: \"kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw\") pod \"maas-api-revoke-sub-3e31a506-gvcxt\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.151761 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.151733 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:35.622322 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:35:35.622275 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ff46e6_8d0f_4320_ae14_e2b777bc4750.slice/crio-1817b3a8db6231f65751a2c4fc7ecfe740ba0ca91fdbca67775ff3f2555aee38 WatchSource:0}: Error finding container 1817b3a8db6231f65751a2c4fc7ecfe740ba0ca91fdbca67775ff3f2555aee38: Status 404 returned error can't find the container with id 1817b3a8db6231f65751a2c4fc7ecfe740ba0ca91fdbca67775ff3f2555aee38 Oct 30 06:35:35.627985 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.626793 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt"] Oct 30 06:35:35.719549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.718624 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-bbbvd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:35:35.719549 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.718700 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-bbbvd" podUID="1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:35:35.751717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:35.751661 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" event={"ID":"08ff46e6-8d0f-4320-ae14-e2b777bc4750","Type":"ContainerStarted","Data":"1817b3a8db6231f65751a2c4fc7ecfe740ba0ca91fdbca67775ff3f2555aee38"} Oct 30 06:35:36.758193 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:36.758163 5565 generic.go:356] "Generic (PLEG): container finished" podID="08ff46e6-8d0f-4320-ae14-e2b777bc4750" containerID="09f34ef765be3a524ba30b71a6b5785535329280b8726c6f5e840cdcdf210824" exitCode=0 Oct 30 06:35:36.758511 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:36.758199 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" event={"ID":"08ff46e6-8d0f-4320-ae14-e2b777bc4750","Type":"ContainerDied","Data":"09f34ef765be3a524ba30b71a6b5785535329280b8726c6f5e840cdcdf210824"} Oct 30 06:35:37.894263 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:37.894234 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:38.065145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.065077 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca\") pod \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " Oct 30 06:35:38.065293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.065155 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca\") pod \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " Oct 30 06:35:38.065293 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.065221 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw\" (UniqueName: \"kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw\") pod \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\" (UID: \"08ff46e6-8d0f-4320-ae14-e2b777bc4750\") " Oct 30 06:35:38.065432 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.065412 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca" pod "08ff46e6-8d0f-4320-ae14-e2b777bc4750" (UID: "08ff46e6-8d0f-4320-ae14-e2b777bc4750"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:35:38.067241 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.067216 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca" pod "08ff46e6-8d0f-4320-ae14-e2b777bc4750" (UID: "08ff46e6-8d0f-4320-ae14-e2b777bc4750"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:38.067365 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.067289 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw" pod "08ff46e6-8d0f-4320-ae14-e2b777bc4750" (UID: "08ff46e6-8d0f-4320-ae14-e2b777bc4750"). InnerVolumeSpecName "kube-api-access-m95lw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:38.166046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.166021 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:38.166046 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.166047 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ff46e6-8d0f-4320-ae14-e2b777bc4750-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:38.166222 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.166064 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m95lw\" (UniqueName: \"kubernetes.io/projected/08ff46e6-8d0f-4320-ae14-e2b777bc4750-kube-api-access-m95lw\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:38.768029 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.767998 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" Oct 30 06:35:38.768205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.768024 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-gvcxt" event={"ID":"08ff46e6-8d0f-4320-ae14-e2b777bc4750","Type":"ContainerDied","Data":"1817b3a8db6231f65751a2c4fc7ecfe740ba0ca91fdbca67775ff3f2555aee38"} Oct 30 06:35:38.768205 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:38.768074 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1817b3a8db6231f65751a2c4fc7ecfe740ba0ca91fdbca67775ff3f2555aee38" Oct 30 06:35:40.104777 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:40.104746 5565 scope.go:122] "RemoveContainer" containerID="818f96474e77edaee20eff1bc7d3c0f1d1b6324c2ad0bb9a184dd11ee4ece21e" Oct 30 06:35:40.113427 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:40.113400 5565 scope.go:122] "RemoveContainer" containerID="4c7a9602dc4ac559f84fbab4f4696de09d72084812a0ce9920be57490f384e5f" Oct 30 06:35:40.121128 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:40.121097 5565 scope.go:122] "RemoveContainer" containerID="d59dfba6147cdb2dbffbe12e773feca87a5fc8074915d2b575cd5210b9b2f20f" Oct 30 06:35:43.389240 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.389205 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" podUID="3e165a8c-0f60-49e6-ac8b-5f9912812892" containerName="main" containerID="cri-o://0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087" gracePeriod=45 Oct 30 06:35:43.464629 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.464601 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" podUID="3e165a8c-0f60-49e6-ac8b-5f9912812892" containerName="main" probeResult="failure" output="Get \"https://10.134.1.71:8000/ready\": dial tcp 10.134.1.71:8000: connect: connection refused" Oct 30 06:35:43.646681 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.646624 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:35:43.713027 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.712989 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home\") pod \"3e165a8c-0f60-49e6-ac8b-5f9912812892\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " Oct 30 06:35:43.713168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713046 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache\") pod \"3e165a8c-0f60-49e6-ac8b-5f9912812892\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " Oct 30 06:35:43.713168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713064 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs\") pod \"3e165a8c-0f60-49e6-ac8b-5f9912812892\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " Oct 30 06:35:43.713168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713086 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir\") pod \"3e165a8c-0f60-49e6-ac8b-5f9912812892\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " Oct 30 06:35:43.713168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713113 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm\") pod \"3e165a8c-0f60-49e6-ac8b-5f9912812892\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " Oct 30 06:35:43.713168 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713135 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl\" (UniqueName: \"kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl\") pod \"3e165a8c-0f60-49e6-ac8b-5f9912812892\" (UID: \"3e165a8c-0f60-49e6-ac8b-5f9912812892\") " Oct 30 06:35:43.713476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713282 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home" pod "3e165a8c-0f60-49e6-ac8b-5f9912812892" (UID: "3e165a8c-0f60-49e6-ac8b-5f9912812892"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:43.713476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713336 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache" pod "3e165a8c-0f60-49e6-ac8b-5f9912812892" (UID: "3e165a8c-0f60-49e6-ac8b-5f9912812892"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:43.713476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713371 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir" pod "3e165a8c-0f60-49e6-ac8b-5f9912812892" (UID: "3e165a8c-0f60-49e6-ac8b-5f9912812892"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:43.713476 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713475 5565 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-home\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:43.713737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713489 5565 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-model-cache\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:43.713737 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.713498 5565 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-tmp-dir\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:43.715410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.715383 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm" pod "3e165a8c-0f60-49e6-ac8b-5f9912812892" (UID: "3e165a8c-0f60-49e6-ac8b-5f9912812892"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 06:35:43.715410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.715398 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl" pod "3e165a8c-0f60-49e6-ac8b-5f9912812892" (UID: "3e165a8c-0f60-49e6-ac8b-5f9912812892"). InnerVolumeSpecName "kube-api-access-txpzl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:35:43.715410 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.715393 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs" pod "3e165a8c-0f60-49e6-ac8b-5f9912812892" (UID: "3e165a8c-0f60-49e6-ac8b-5f9912812892"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:35:43.791679 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.791653 5565 generic.go:356] "Generic (PLEG): container finished" podID="3e165a8c-0f60-49e6-ac8b-5f9912812892" containerID="0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087" exitCode=0 Oct 30 06:35:43.791796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.791697 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" Oct 30 06:35:43.791796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.791745 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" event={"ID":"3e165a8c-0f60-49e6-ac8b-5f9912812892","Type":"ContainerDied","Data":"0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087"} Oct 30 06:35:43.791796 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.791792 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh" event={"ID":"3e165a8c-0f60-49e6-ac8b-5f9912812892","Type":"ContainerDied","Data":"a91d503f0258c2c4a061a8492a5df4ed4f1dda0d3c168c6c4b38e2497bcf55e7"} Oct 30 06:35:43.791968 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.791817 5565 scope.go:122] "RemoveContainer" containerID="0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087" Oct 30 06:35:43.800998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.800983 5565 scope.go:122] "RemoveContainer" containerID="0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087" Oct 30 06:35:43.801255 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:35:43.801237 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087\": container with ID starting with 0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087 not found: ID does not exist" containerID="0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087" Oct 30 06:35:43.801306 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.801262 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087"} err="failed to get container status \"0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087\": rpc error: code = NotFound desc = could not find container \"0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087\": container with ID starting with 0da4246612a1806cf4ffa10ad1ab93c0cc39235d3f15cee532132b911338b087 not found: ID does not exist" Oct 30 06:35:43.814191 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.814168 5565 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e165a8c-0f60-49e6-ac8b-5f9912812892-tls-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:43.814191 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.814187 5565 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e165a8c-0f60-49e6-ac8b-5f9912812892-dshm\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:43.814322 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.814196 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txpzl\" (UniqueName: \"kubernetes.io/projected/3e165a8c-0f60-49e6-ac8b-5f9912812892-kube-api-access-txpzl\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:35:43.848189 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.848161 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh"] Oct 30 06:35:43.852357 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:43.852330 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-b658b769-kserve-7746468b57-r5vkh"] Oct 30 06:35:45.572243 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:45.567623 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e165a8c-0f60-49e6-ac8b-5f9912812892" path="/var/lib/kubelet/pods/3e165a8c-0f60-49e6-ac8b-5f9912812892/volumes" Oct 30 06:35:45.949998 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:45.949903 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:35:47.638019 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:35:47.637982 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:35:53.181781 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:53.181752 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:35:58.249124 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249088 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x"] Oct 30 06:35:58.249689 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249670 5565 cpu_manager.go:414] "containerMap: removing container" podUID="3e165a8c-0f60-49e6-ac8b-5f9912812892" containerName="main" Oct 30 06:35:58.249790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249691 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e165a8c-0f60-49e6-ac8b-5f9912812892" containerName="main" Oct 30 06:35:58.249790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249743 5565 cpu_manager.go:414] "containerMap: removing container" podUID="08ff46e6-8d0f-4320-ae14-e2b777bc4750" containerName="revoke-keys" Oct 30 06:35:58.249790 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249754 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ff46e6-8d0f-4320-ae14-e2b777bc4750" containerName="revoke-keys" Oct 30 06:35:58.249951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249855 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ff46e6-8d0f-4320-ae14-e2b777bc4750" containerName="revoke-keys" Oct 30 06:35:58.249951 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.249874 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e165a8c-0f60-49e6-ac8b-5f9912812892" containerName="main" Oct 30 06:35:58.254502 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.254481 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.286649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.286622 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:35:58.286649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.286638 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-fvgnj\"" Oct 30 06:35:58.286649 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.286655 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 06:35:58.286864 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.286664 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:35:58.288229 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.288211 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x"] Oct 30 06:35:58.291207 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.291187 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.291341 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.291322 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.291387 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.291352 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqmws\" (UniqueName: \"kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.392266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.392239 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.392266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.392268 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqmws\" (UniqueName: \"kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.392495 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.392296 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.392935 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.392911 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.394517 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.394499 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.417076 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.417052 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqmws\" (UniqueName: \"kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws\") pod \"maas-api-revoke-sub-3e4fd46e-nwb8x\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.566955 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.566927 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:35:58.865795 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.865112 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x"] Oct 30 06:35:58.873060 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:35:58.873022 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a53be76_25e9_49a6_a808_faaa92ac5468.slice/crio-d8a05b5cf3f96aabbdfb17ff578a1b72d1cd892b5aa6d700a2d54f757000b279 WatchSource:0}: Error finding container d8a05b5cf3f96aabbdfb17ff578a1b72d1cd892b5aa6d700a2d54f757000b279: Status 404 returned error can't find the container with id d8a05b5cf3f96aabbdfb17ff578a1b72d1cd892b5aa6d700a2d54f757000b279 Oct 30 06:35:58.879886 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:58.879861 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" event={"ID":"0a53be76-25e9-49a6-a808-faaa92ac5468","Type":"ContainerStarted","Data":"d8a05b5cf3f96aabbdfb17ff578a1b72d1cd892b5aa6d700a2d54f757000b279"} Oct 30 06:35:59.885859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:59.885784 5565 generic.go:356] "Generic (PLEG): container finished" podID="0a53be76-25e9-49a6-a808-faaa92ac5468" containerID="c44a072ff23c53c9422af4f7707b7e67baf2cca760c6bf27caf22ec015e3f61b" exitCode=0 Oct 30 06:35:59.885859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:35:59.885834 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" event={"ID":"0a53be76-25e9-49a6-a808-faaa92ac5468","Type":"ContainerDied","Data":"c44a072ff23c53c9422af4f7707b7e67baf2cca760c6bf27caf22ec015e3f61b"} Oct 30 06:36:01.032747 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.032708 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:36:01.113793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.113749 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca\") pod \"0a53be76-25e9-49a6-a808-faaa92ac5468\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " Oct 30 06:36:01.114018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.113809 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws\" (UniqueName: \"kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws\") pod \"0a53be76-25e9-49a6-a808-faaa92ac5468\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " Oct 30 06:36:01.114018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.113840 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca\") pod \"0a53be76-25e9-49a6-a808-faaa92ac5468\" (UID: \"0a53be76-25e9-49a6-a808-faaa92ac5468\") " Oct 30 06:36:01.114234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.114205 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca" pod "0a53be76-25e9-49a6-a808-faaa92ac5468" (UID: "0a53be76-25e9-49a6-a808-faaa92ac5468"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:01.116165 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.116126 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws" pod "0a53be76-25e9-49a6-a808-faaa92ac5468" (UID: "0a53be76-25e9-49a6-a808-faaa92ac5468"). InnerVolumeSpecName "kube-api-access-zqmws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:01.116312 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.116187 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca" pod "0a53be76-25e9-49a6-a808-faaa92ac5468" (UID: "0a53be76-25e9-49a6-a808-faaa92ac5468"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:01.214922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.214833 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-openshift-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:01.214922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.214865 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqmws\" (UniqueName: \"kubernetes.io/projected/0a53be76-25e9-49a6-a808-faaa92ac5468-kube-api-access-zqmws\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:01.214922 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.214875 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0a53be76-25e9-49a6-a808-faaa92ac5468-maas-api-xks-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:01.896267 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.896181 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" Oct 30 06:36:01.896434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.896181 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e4fd46e-nwb8x" event={"ID":"0a53be76-25e9-49a6-a808-faaa92ac5468","Type":"ContainerDied","Data":"d8a05b5cf3f96aabbdfb17ff578a1b72d1cd892b5aa6d700a2d54f757000b279"} Oct 30 06:36:01.896434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:01.896311 5565 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d8a05b5cf3f96aabbdfb17ff578a1b72d1cd892b5aa6d700a2d54f757000b279" Oct 30 06:36:09.370847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:09.370807 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:36:10.525424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.525389 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr"] Oct 30 06:36:10.525847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.525814 5565 cpu_manager.go:414] "containerMap: removing container" podUID="0a53be76-25e9-49a6-a808-faaa92ac5468" containerName="revoke-keys" Oct 30 06:36:10.525847 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.525826 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a53be76-25e9-49a6-a808-faaa92ac5468" containerName="revoke-keys" Oct 30 06:36:10.525959 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.525948 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a53be76-25e9-49a6-a808-faaa92ac5468" containerName="revoke-keys" Oct 30 06:36:10.529068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.529040 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.546858 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.546835 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 06:36:10.560896 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.560860 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 06:36:10.562635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.562611 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 06:36:10.562769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.562685 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 06:36:10.562769 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.562754 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7wbd8\"" Oct 30 06:36:10.562853 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.562775 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr"] Oct 30 06:36:10.606340 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.606313 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.606512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.606375 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.606512 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.606404 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.606662 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.606601 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnrdc\" (UniqueName: \"kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.707319 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.707294 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnrdc\" (UniqueName: \"kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.707482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.707355 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.707482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.707405 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.707482 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.707427 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.708195 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.708167 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.709895 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.709875 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.710042 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.710022 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.731877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.731851 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnrdc\" (UniqueName: \"kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc\") pod \"maas-api-5c48877c75-8c9kr\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:10.842696 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:10.842632 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:11.143036 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:11.142957 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr"] Oct 30 06:36:11.145369 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:36:11.145337 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557c7bc1_53e3_48bb_bfa2_55daba6ce609.slice/crio-fba3344ef45f0bd54a6aed0f642f75701159b469fa258a4400f1f8755516b49a WatchSource:0}: Error finding container fba3344ef45f0bd54a6aed0f642f75701159b469fa258a4400f1f8755516b49a: Status 404 returned error can't find the container with id fba3344ef45f0bd54a6aed0f642f75701159b469fa258a4400f1f8755516b49a Oct 30 06:36:11.146688 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:11.146668 5565 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 06:36:11.949058 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:11.949026 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" event={"ID":"557c7bc1-53e3-48bb-bfa2-55daba6ce609","Type":"ContainerStarted","Data":"18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005"} Oct 30 06:36:11.949451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:11.949100 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" event={"ID":"557c7bc1-53e3-48bb-bfa2-55daba6ce609","Type":"ContainerStarted","Data":"fba3344ef45f0bd54a6aed0f642f75701159b469fa258a4400f1f8755516b49a"} Oct 30 06:36:11.949451 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:11.949146 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:11.995656 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:11.995611 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" podStartSLOduration=1.6826407620000001 podStartE2EDuration="1.995599384s" totalImagesPullingTime="312.958622ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:11.994056642 +0000 UTC m=+3878.569743696" watchObservedRunningTime="2026-10-30 06:36:11.995599384 +0000 UTC m=+3878.571286438" Oct 30 06:36:12.034651 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:12.034622 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr"] Oct 30 06:36:12.036859 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:12.036838 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qhggr"] Oct 30 06:36:13.566518 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:13.566472 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb47ec55-40f3-45ae-b3a1-754002b94c19" path="/var/lib/kubelet/pods/bb47ec55-40f3-45ae-b3a1-754002b94c19/volumes" Oct 30 06:36:17.531363 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:36:17.531312 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:36:17.959699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:17.959620 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:18.789438 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.789406 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rmzbk"] Oct 30 06:36:18.793234 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.793216 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.814905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.814877 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rmzbk"] Oct 30 06:36:18.886717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.886686 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-maas-api-tls\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.886867 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.886728 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-metrics-certs\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.886916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.886868 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nnqj\" (UniqueName: \"kubernetes.io/projected/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-kube-api-access-9nnqj\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.886916 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.886905 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-openshift-service-ca\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.988160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.988126 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nnqj\" (UniqueName: \"kubernetes.io/projected/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-kube-api-access-9nnqj\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.988160 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.988160 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-openshift-service-ca\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.988371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.988201 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-maas-api-tls\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.988371 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.988225 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-metrics-certs\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.988882 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.988860 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-openshift-service-ca\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.990766 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.990742 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-maas-api-tls\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:18.990869 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:18.990793 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-metrics-certs\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:19.013000 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.012978 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nnqj\" (UniqueName: \"kubernetes.io/projected/b8640cfd-ebc9-4fa3-a941-bc90ad425a3e-kube-api-access-9nnqj\") pod \"maas-api-595cc648c-rmzbk\" (UID: \"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:19.085960 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.085884 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:36:19.105898 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.105871 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:19.410692 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.410665 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-rmzbk"] Oct 30 06:36:19.412618 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:36:19.412594 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8640cfd_ebc9_4fa3_a941_bc90ad425a3e.slice/crio-8819a7ab3e576fd07be719cafd0fc6163fba507cc8bcc0afb80d05e640666102 WatchSource:0}: Error finding container 8819a7ab3e576fd07be719cafd0fc6163fba507cc8bcc0afb80d05e640666102: Status 404 returned error can't find the container with id 8819a7ab3e576fd07be719cafd0fc6163fba507cc8bcc0afb80d05e640666102 Oct 30 06:36:19.993492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.993441 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" event={"ID":"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e","Type":"ContainerStarted","Data":"1827b811320af4bc50f110ac2773deb28f3ddde7b0734dc11d7136117a42e2ae"} Oct 30 06:36:19.993964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.993516 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" event={"ID":"b8640cfd-ebc9-4fa3-a941-bc90ad425a3e","Type":"ContainerStarted","Data":"8819a7ab3e576fd07be719cafd0fc6163fba507cc8bcc0afb80d05e640666102"} Oct 30 06:36:19.993964 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:19.993573 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:20.039662 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:20.039611 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" podStartSLOduration=1.588001507 podStartE2EDuration="2.0395978s" totalImagesPullingTime="451.596293ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 06:36:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:36:20.038932738 +0000 UTC m=+3886.614619817" watchObservedRunningTime="2026-10-30 06:36:20.0395978 +0000 UTC m=+3886.615284851" Oct 30 06:36:26.023624 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.023596 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-rmzbk" Oct 30 06:36:26.104043 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.104010 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr"] Oct 30 06:36:26.104247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.104221 5565 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" podUID="557c7bc1-53e3-48bb-bfa2-55daba6ce609" containerName="maas-api" containerID="cri-o://18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005" gracePeriod=30 Oct 30 06:36:26.343305 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.343283 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:26.386019 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.385995 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls\") pod \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " Oct 30 06:36:26.386103 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.386051 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca\") pod \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " Oct 30 06:36:26.386155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.386114 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs\") pod \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " Oct 30 06:36:26.386223 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.386200 5565 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc\" (UniqueName: \"kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc\") pod \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\" (UID: \"557c7bc1-53e3-48bb-bfa2-55daba6ce609\") " Oct 30 06:36:26.386466 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.386435 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca" pod "557c7bc1-53e3-48bb-bfa2-55daba6ce609" (UID: "557c7bc1-53e3-48bb-bfa2-55daba6ce609"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 06:36:26.386645 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.386627 5565 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/557c7bc1-53e3-48bb-bfa2-55daba6ce609-openshift-service-ca\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:26.388145 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.388118 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls" pod "557c7bc1-53e3-48bb-bfa2-55daba6ce609" (UID: "557c7bc1-53e3-48bb-bfa2-55daba6ce609"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:26.388266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.388155 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs" pod "557c7bc1-53e3-48bb-bfa2-55daba6ce609" (UID: "557c7bc1-53e3-48bb-bfa2-55daba6ce609"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 06:36:26.388266 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.388225 5565 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc" pod "557c7bc1-53e3-48bb-bfa2-55daba6ce609" (UID: "557c7bc1-53e3-48bb-bfa2-55daba6ce609"). InnerVolumeSpecName "kube-api-access-qnrdc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 06:36:26.488064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.488036 5565 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-maas-api-tls\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:26.488064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.488059 5565 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/557c7bc1-53e3-48bb-bfa2-55daba6ce609-metrics-certs\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:26.488064 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:26.488069 5565 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnrdc\" (UniqueName: \"kubernetes.io/projected/557c7bc1-53e3-48bb-bfa2-55daba6ce609-kube-api-access-qnrdc\") on node \"ip-10-0-129-175.ec2.internal\" DevicePath \"\"" Oct 30 06:36:27.045821 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.045788 5565 generic.go:356] "Generic (PLEG): container finished" podID="557c7bc1-53e3-48bb-bfa2-55daba6ce609" containerID="18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005" exitCode=0 Oct 30 06:36:27.046332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.045841 5565 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" Oct 30 06:36:27.046332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.045860 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" event={"ID":"557c7bc1-53e3-48bb-bfa2-55daba6ce609","Type":"ContainerDied","Data":"18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005"} Oct 30 06:36:27.046332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.045899 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr" event={"ID":"557c7bc1-53e3-48bb-bfa2-55daba6ce609","Type":"ContainerDied","Data":"fba3344ef45f0bd54a6aed0f642f75701159b469fa258a4400f1f8755516b49a"} Oct 30 06:36:27.046332 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.045919 5565 scope.go:122] "RemoveContainer" containerID="18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005" Oct 30 06:36:27.055409 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.055391 5565 scope.go:122] "RemoveContainer" containerID="18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005" Oct 30 06:36:27.055689 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:36:27.055669 5565 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005\": container with ID starting with 18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005 not found: ID does not exist" containerID="18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005" Oct 30 06:36:27.055744 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.055698 5565 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005"} err="failed to get container status \"18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005\": rpc error: code = NotFound desc = could not find container \"18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005\": container with ID starting with 18c7e4eddac760fa9be39c1e3ddb36d27d3cb80c67129371e4ca867ed0ee5005 not found: ID does not exist" Oct 30 06:36:27.105500 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.105472 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr"] Oct 30 06:36:27.118732 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.118706 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-5c48877c75-8c9kr"] Oct 30 06:36:27.251635 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.251612 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:36:27.565792 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:27.565763 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="557c7bc1-53e3-48bb-bfa2-55daba6ce609" path="/var/lib/kubelet/pods/557c7bc1-53e3-48bb-bfa2-55daba6ce609/volumes" Oct 30 06:36:41.183143 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:36:41.183101 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:36:47.534238 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:36:47.534194 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:37:09.058142 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:09.058110 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-ct2m7"] Oct 30 06:37:09.493927 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:09.493839 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7549c48cc4-skclp_d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f/authorino/0.log" Oct 30 06:37:10.676684 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:10.676622 5565 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-84587756dc-6kkhz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 06:37:10.678348 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:10.677969 5565 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-84587756dc-6kkhz" podUID="dcf9bedc-ab39-4ca4-a56b-21db78fb6443" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.54:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 06:37:17.532096 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:37:17.532057 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:37:20.065849 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:20.065813 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-554787856d-zdmnb_6eca8a08-a9dc-45b8-8331-4751c3bc20b7/manager/0.log" Oct 30 06:37:22.212779 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:22.212742 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh_b79bb003-300f-46d9-b5e3-3a3289d1aee9/manager/0.log" Oct 30 06:37:22.246963 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:22.246936 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh_b79bb003-300f-46d9-b5e3-3a3289d1aee9/copy-manifests/0.log" Oct 30 06:37:23.695291 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:23.695254 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-rmzbk_b8640cfd-ebc9-4fa3-a941-bc90ad425a3e/maas-api/0.log" Oct 30 06:37:24.079020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:24.078989 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gvcxt_08ff46e6-8d0f-4320-ae14-e2b777bc4750/revoke-keys/0.log" Oct 30 06:37:24.260593 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:24.260568 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e4fd46e-nwb8x_0a53be76-25e9-49a6-a808-faaa92ac5468/revoke-keys/0.log" Oct 30 06:37:24.447932 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:24.447854 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-zvd7w_7d6712b1-9033-465b-b9c1-e45f393de1ce/revoke-keys/0.log" Oct 30 06:37:25.015386 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:25.015353 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4hmbr_6526d6e7-6777-4d47-abce-81f48bb08a4f/revoke-keys/0.log" Oct 30 06:37:25.192576 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:25.192545 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-czpj9_7bd49e77-7489-4696-a6be-46882366d99d/revoke-keys/0.log" Oct 30 06:37:27.059492 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:27.059457 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7549c48cc4-skclp_d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f/authorino/0.log" Oct 30 06:37:27.305001 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:27.304973 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-pwttx_db0127c0-1218-470b-9769-4ae07d0b80ff/manager/0.log" Oct 30 06:37:28.195758 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:28.195732 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ct2m7_7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f/limitador/0.log" Oct 30 06:37:29.585246 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:29.585215 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788bd9946-f8sxq_80ec3dc9-bb84-47af-89d2-6db6ffe9df05/kube-auth-proxy/0.log" Oct 30 06:37:29.964217 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:29.964140 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-6kkhz_dcf9bedc-ab39-4ca4-a56b-21db78fb6443/istio-proxy/0.log" Oct 30 06:37:30.593764 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:30.593728 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-bbbvd_1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20/istio-proxy/0.log" Oct 30 06:37:31.761580 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:31.761518 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-xp2kr_8611d5b8-cd8b-4117-b571-b96f2cde2621/istio-proxy/0.log" Oct 30 06:37:32.141434 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:32.141353 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:37:33.703208 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:33.703176 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9hmzl_e413ad81-bb52-45f7-86c4-7127a1564739/main/0.log" Oct 30 06:37:34.109439 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:34.109404 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5_930df857-6e36-4c04-a098-ccec27be1dec/main/0.log" Oct 30 06:37:34.309967 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:34.309937 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2_8025876f-d78e-4475-afd3-4ba65437847f/main/0.log" Oct 30 06:37:35.196768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:35.196734 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7549c48cc4-skclp_d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f/authorino/0.log" Oct 30 06:37:40.172822 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:40.172789 5565 scope.go:122] "RemoveContainer" containerID="8b80756cebfe82fedb3b5d818bb1ed6172f284a2bc1d34711366b76de0d65bde" Oct 30 06:37:44.155908 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:44.155869 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_ai-gateway-controller-554787856d-zdmnb_6eca8a08-a9dc-45b8-8331-4751c3bc20b7/manager/0.log" Oct 30 06:37:46.130041 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:46.129968 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh_b79bb003-300f-46d9-b5e3-3a3289d1aee9/copy-manifests/0.log" Oct 30 06:37:46.160171 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:46.160143 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6c6b8df5dd-dg8kh_b79bb003-300f-46d9-b5e3-3a3289d1aee9/manager/0.log" Oct 30 06:37:47.353713 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:47.353636 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-rmzbk_b8640cfd-ebc9-4fa3-a941-bc90ad425a3e/maas-api/0.log" Oct 30 06:37:47.533260 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:37:47.533226 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:37:47.739903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:47.739826 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-gvcxt_08ff46e6-8d0f-4320-ae14-e2b777bc4750/revoke-keys/0.log" Oct 30 06:37:47.922125 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:47.922093 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e4fd46e-nwb8x_0a53be76-25e9-49a6-a808-faaa92ac5468/revoke-keys/0.log" Oct 30 06:37:48.098953 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:48.098926 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-zvd7w_7d6712b1-9033-465b-b9c1-e45f393de1ce/revoke-keys/0.log" Oct 30 06:37:48.651130 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:48.651103 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d2bc93bc-4hmbr_6526d6e7-6777-4d47-abce-81f48bb08a4f/revoke-keys/0.log" Oct 30 06:37:48.830489 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:48.830466 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d576e68b-czpj9_7bd49e77-7489-4696-a6be-46882366d99d/revoke-keys/0.log" Oct 30 06:37:50.301388 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:50.301359 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7549c48cc4-skclp_d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f/authorino/0.log" Oct 30 06:37:50.546424 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:50.546393 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-pwttx_db0127c0-1218-470b-9769-4ae07d0b80ff/manager/0.log" Oct 30 06:37:51.377548 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:51.377499 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ct2m7_7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f/limitador/0.log" Oct 30 06:37:52.501877 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:52.501852 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788bd9946-f8sxq_80ec3dc9-bb84-47af-89d2-6db6ffe9df05/kube-auth-proxy/0.log" Oct 30 06:37:52.865454 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:52.865422 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-6kkhz_dcf9bedc-ab39-4ca4-a56b-21db78fb6443/istio-proxy/0.log" Oct 30 06:37:53.485559 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:53.485519 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-bbbvd_1d63c4f9-cde5-4b64-a152-9e8a6cb0ee20/istio-proxy/0.log" Oct 30 06:37:54.635018 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:54.634982 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-xp2kr_8611d5b8-cd8b-4117-b571-b96f2cde2621/istio-proxy/0.log" Oct 30 06:37:55.013264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:55.013235 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:37:56.330599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:56.330566 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-9hmzl_e413ad81-bb52-45f7-86c4-7127a1564739/main/0.log" Oct 30 06:37:56.719155 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:56.719078 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-s8jc5_930df857-6e36-4c04-a098-ccec27be1dec/main/0.log" Oct 30 06:37:56.918247 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:37:56.918222 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-w7dv2_8025876f-d78e-4475-afd3-4ba65437847f/main/0.log" Oct 30 06:38:02.046483 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:02.046452 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9"] Oct 30 06:38:02.059873 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:02.059848 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-czpj9"] Oct 30 06:38:03.567863 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:03.567829 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd49e77-7489-4696-a6be-46882366d99d" path="/var/lib/kubelet/pods/7bd49e77-7489-4696-a6be-46882366d99d/volumes" Oct 30 06:38:17.531401 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:38:17.531358 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:38:19.030302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:19.030266 5565 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr"] Oct 30 06:38:19.044057 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:19.044028 5565 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-4hmbr"] Oct 30 06:38:19.565302 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:19.565267 5565 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6526d6e7-6777-4d47-abce-81f48bb08a4f" path="/var/lib/kubelet/pods/6526d6e7-6777-4d47-abce-81f48bb08a4f/volumes" Oct 30 06:38:30.817521 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:30.817491 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:38:31.013702 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:38:31.013673 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:38:47.531991 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:38:47.531944 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:39:13.600122 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:13.600095 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:39:13.786108 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:13.786082 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5f78687d5b-m62wv_3b17179d-de10-4330-a48a-e5aa6ecca579/payload-processing/0.log" Oct 30 06:39:17.531665 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:39:17.531587 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:39:19.091980 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:19.091951 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rzb85_2e463762-df03-4b5c-be5f-19892ed3d24b/global-pull-secret-syncer/0.log" Oct 30 06:39:19.862587 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:19.862555 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-lh482_eddd45fc-666b-4ba0-a049-10623018ea60/konnectivity-agent/0.log" Oct 30 06:39:20.123871 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:20.123802 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-175.ec2.internal_44ac8e1d4fed32053c2639c09d7471d6/haproxy/0.log" Oct 30 06:39:26.431264 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:26.431226 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7549c48cc4-skclp_d32ee48f-6a9f-4c44-8a8b-f5baaf0a416f/authorino/0.log" Oct 30 06:39:26.528946 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:26.528914 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-68d6dd76dc-pwttx_db0127c0-1218-470b-9769-4ae07d0b80ff/manager/0.log" Oct 30 06:39:27.063468 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:27.063434 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-ct2m7_7c62a87f-5f27-403b-8cf9-e2a71b2dfd6f/limitador/0.log" Oct 30 06:39:28.538311 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.538281 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/alertmanager/0.log" Oct 30 06:39:28.653818 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.653794 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/config-reloader/0.log" Oct 30 06:39:28.773746 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.773713 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/kube-rbac-proxy-web/0.log" Oct 30 06:39:28.815547 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.815438 5565 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h"] Oct 30 06:39:28.815903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.815887 5565 cpu_manager.go:414] "containerMap: removing container" podUID="557c7bc1-53e3-48bb-bfa2-55daba6ce609" containerName="maas-api" Oct 30 06:39:28.815903 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.815904 5565 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="557c7bc1-53e3-48bb-bfa2-55daba6ce609" containerName="maas-api" Oct 30 06:39:28.816005 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.815995 5565 memory_manager.go:365] "RemoveStaleState removing state" podUID="557c7bc1-53e3-48bb-bfa2-55daba6ce609" containerName="maas-api" Oct 30 06:39:28.819068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.819048 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:28.835296 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.835278 5565 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-c66wq\"/\"default-dockercfg-vlpb2\"" Oct 30 06:39:28.850831 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.850809 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h"] Oct 30 06:39:28.895068 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.895043 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/kube-rbac-proxy/0.log" Oct 30 06:39:28.910687 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.910660 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbt4f\" (UniqueName: \"kubernetes.io/projected/3fcf9634-16c6-4750-ace8-b8037db557a2-kube-api-access-xbt4f\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:28.910788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.910768 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-lib-modules\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:28.910855 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.910813 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-sys\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:28.910917 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.910882 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-podres\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:28.910976 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.910925 5565 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-proc\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:28.935625 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.935599 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c66wq\"/\"openshift-service-ca.crt\"" Oct 30 06:39:28.935717 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:28.935629 5565 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-c66wq\"/\"kube-root-ca.crt\"" Oct 30 06:39:29.010391 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.010364 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/kube-rbac-proxy-metric/0.log" Oct 30 06:39:29.011797 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011779 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-lib-modules\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.011862 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011813 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-sys\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.011905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011854 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-podres\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.011905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011881 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-sys\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.011905 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011888 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-proc\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.012020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011936 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-lib-modules\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.012020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011948 5565 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbt4f\" (UniqueName: \"kubernetes.io/projected/3fcf9634-16c6-4750-ace8-b8037db557a2-kube-api-access-xbt4f\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.012020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.011971 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-podres\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.012020 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.012001 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3fcf9634-16c6-4750-ace8-b8037db557a2-proc\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.037384 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.037359 5565 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbt4f\" (UniqueName: \"kubernetes.io/projected/3fcf9634-16c6-4750-ace8-b8037db557a2-kube-api-access-xbt4f\") pod \"perf-node-gather-daemonset-dcd6h\" (UID: \"3fcf9634-16c6-4750-ace8-b8037db557a2\") " pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.125071 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.125007 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/prom-label-proxy/0.log" Oct 30 06:39:29.131094 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.131072 5565 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:29.243479 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.243451 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_0073ab4b-bf58-4c99-81d6-049a441f9d96/init-config-reloader/0.log" Oct 30 06:39:29.429983 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.429947 5565 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h"] Oct 30 06:39:29.432352 ip-10-0-129-175 kubenswrapper[5565]: W1030 06:39:29.432313 5565 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3fcf9634_16c6_4750_ace8_b8037db557a2.slice/crio-c666051d1670213a0abf2aeafe491f5df6f4bb9c4bdde7ba1c4c5b0534ce0040 WatchSource:0}: Error finding container c666051d1670213a0abf2aeafe491f5df6f4bb9c4bdde7ba1c4c5b0534ce0040: Status 404 returned error can't find the container with id c666051d1670213a0abf2aeafe491f5df6f4bb9c4bdde7ba1c4c5b0534ce0040 Oct 30 06:39:29.858883 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.858855 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6b8f6c9d6d-ttlgm_fdffe71b-09d7-4f44-9d2e-45cae3f24023/metrics-server/0.log" Oct 30 06:39:29.976721 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:29.976690 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-kdkbs_72b235d4-33f4-48d7-946b-e218318547ca/monitoring-plugin/0.log" Oct 30 06:39:30.017568 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:30.017518 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" event={"ID":"3fcf9634-16c6-4750-ace8-b8037db557a2","Type":"ContainerStarted","Data":"0417042cae3ef7f9ba646a91a7510a2fc721b7b01be1dd7ff8a968ff3e55cddb"} Oct 30 06:39:30.017699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:30.017590 5565 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" event={"ID":"3fcf9634-16c6-4750-ace8-b8037db557a2","Type":"ContainerStarted","Data":"c666051d1670213a0abf2aeafe491f5df6f4bb9c4bdde7ba1c4c5b0534ce0040"} Oct 30 06:39:30.017699 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:30.017612 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:30.061729 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:30.061688 5565 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" podStartSLOduration=2.061676687 podStartE2EDuration="2.061676687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 06:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 06:39:30.061204056 +0000 UTC m=+4076.636891111" watchObservedRunningTime="2026-10-30 06:39:30.061676687 +0000 UTC m=+4076.637363760" Oct 30 06:39:30.789793 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:30.789762 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k22vf_bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/node-exporter/0.log" Oct 30 06:39:30.907355 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:30.907332 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k22vf_bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/kube-rbac-proxy/0.log" Oct 30 06:39:31.024753 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:31.024732 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-k22vf_bc3d7732-2c73-44cb-9ec4-f083b1b95a7a/init-textfile/0.log" Oct 30 06:39:31.512308 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:31.512282 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/prometheus/0.log" Oct 30 06:39:31.628870 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:31.628835 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/config-reloader/0.log" Oct 30 06:39:31.744599 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:31.744572 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/thanos-sidecar/0.log" Oct 30 06:39:31.860044 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:31.859975 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/kube-rbac-proxy-web/0.log" Oct 30 06:39:31.982244 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:31.982216 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/kube-rbac-proxy/0.log" Oct 30 06:39:32.096448 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:32.096424 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/kube-rbac-proxy-thanos/0.log" Oct 30 06:39:32.211677 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:32.211604 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_d44bc086-596f-42d2-bfa2-ad0fae67e64a/init-config-reloader/0.log" Oct 30 06:39:32.674585 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:32.674560 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56556f6c49-cqssd_8fa63c73-1e28-4ef7-9672-29a5a36997d9/telemeter-client/0.log" Oct 30 06:39:32.788262 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:32.788238 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56556f6c49-cqssd_8fa63c73-1e28-4ef7-9672-29a5a36997d9/reload/0.log" Oct 30 06:39:32.902091 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:32.902060 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-56556f6c49-cqssd_8fa63c73-1e28-4ef7-9672-29a5a36997d9/kube-rbac-proxy/0.log" Oct 30 06:39:33.022154 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:33.022128 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b554f6f57-vkkxh_cc069a86-752c-499c-98a0-c6ad926307f7/thanos-query/0.log" Oct 30 06:39:33.135605 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:33.135583 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b554f6f57-vkkxh_cc069a86-752c-499c-98a0-c6ad926307f7/kube-rbac-proxy-web/0.log" Oct 30 06:39:33.250544 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:33.250508 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b554f6f57-vkkxh_cc069a86-752c-499c-98a0-c6ad926307f7/kube-rbac-proxy/0.log" Oct 30 06:39:33.365788 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:33.365719 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b554f6f57-vkkxh_cc069a86-752c-499c-98a0-c6ad926307f7/prom-label-proxy/0.log" Oct 30 06:39:33.480984 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:33.480955 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b554f6f57-vkkxh_cc069a86-752c-499c-98a0-c6ad926307f7/kube-rbac-proxy-rules/0.log" Oct 30 06:39:33.601693 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:33.601670 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5b554f6f57-vkkxh_cc069a86-752c-499c-98a0-c6ad926307f7/kube-rbac-proxy-metrics/0.log" Oct 30 06:39:36.034113 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:36.034082 5565 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-c66wq/perf-node-gather-daemonset-dcd6h" Oct 30 06:39:40.200088 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:40.200059 5565 scope.go:122] "RemoveContainer" containerID="b5bcfa4d3af6b142726d3babd88f5047777eca56cfd272673833ad58045d5430" Oct 30 06:39:40.208537 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:40.208504 5565 scope.go:122] "RemoveContainer" containerID="64d0dcc6b4138a1d8c6db11dcfd53952dc132ba3a83095d0f46a903a54ffd843" Oct 30 06:39:40.626768 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:40.626738 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-plfkr_54b31cad-aaef-4ade-adf9-4664d673e085/dns/0.log" Oct 30 06:39:40.744190 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:40.744162 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-plfkr_54b31cad-aaef-4ade-adf9-4664d673e085/kube-rbac-proxy/0.log" Oct 30 06:39:41.099116 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:41.099076 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8wf8z_67e26d01-cd9d-4e17-bd57-f7eb78cdc92e/dns-node-resolver/0.log" Oct 30 06:39:42.597130 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:42.597089 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gbpks_42226e87-0da6-4fa1-8d6c-842368393e3e/node-ca/0.log" Oct 30 06:39:47.532929 ip-10-0-129-175 kubenswrapper[5565]: E1030 06:39:47.532892 5565 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p69qm" containerName="extractor" Oct 30 06:39:47.641104 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:47.641075 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-788bd9946-f8sxq_80ec3dc9-bb84-47af-89d2-6db6ffe9df05/kube-auth-proxy/0.log" Oct 30 06:39:47.784447 ip-10-0-129-175 kubenswrapper[5565]: I1030 06:39:47.784309 5565 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-84587756dc-6kkhz_dcf9bedc-ab39-4ca4-a56b-21db78fb6443/istio-proxy/0.log"